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ABSTRACT 

As a preliminary to developing criteria to meet 
quality control requirements of the National Literacy Act of 1991, 
this report analyzes four, areas where state education agencies 
examine program quality: local program monitoring to assess 
compliance or promote program improvement; criteria used to evaluate 
local program funding applications; program evaluations of state or 
local adult education programs and evaluations of projects designed 
to improve program quality; and explicit indicators of local program 
quality already in use by states. Data were gathered through a review 
of 46 monitoring instruments and 20 proposal review protocols, 
requests for proposals, or funding criteria. Following an 
introductory section describing the research methodology and aims of 
the study, the report is organized in five sections. Section 2 
presents monitoring indicators, section 3 evaluation indicators, and 
section 4 funding indicators. Each section includes descriptions of 
documents reviewed and summaries of research methodology, of how the 
information is used, and of the general topics and quality indicators 
used in a majority of documents under review. The information is 
presented in tabular form by component area, topic area, indicator, 
and specific measure. Section 5 cf the report describes explicit 
quality indicators used in seven states, and the final section lists 
quality indicators synthesized from all four sources. Appendixes, 
which make up half the document, present individual summaries of the 
program evaluation and state summaries of funding criteria. (KC) 
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SYNTHESIS OF STATE QUALITY INDICATORS 
FOR ADULT EDUCATION PROGRAMS 



I. INTRODUCTION 

The National Literacy Act of 1991 requires th* states to develop indicators of program 
quality for adult education programs by July 1993. To assist states with this process, the Act also 
requires the Department of Education (ED) to develop by July 1992 model indicators of program 
quality as guidance to states. The indicators are to be developed through consultation with 
experts, educators and administrators of adult education. To rreet :his requirement, the Office of 
Vocational and Adult Education (OVAE) will convene four working groups of individuals 
representing these constituencies who will prepare a preliminary list of indicators for OVAE to 
consider. 

To assist the working groups, OVAE contracted with Pelavin Associates to conduct a 
background review and synthesis of indicators of program quality being used by state education 
agencies (SEAs) and researchers. This review will also assist OVAE and SEAs as they consider 
potential indicators. The purpose of this report is to provide a comprehensive picture of how 
states currently conceptualize and measure program quality and thereby empirically derive 
preliminary indicators as a starting point for the working groups. The . port examines four areas 
where SEAs explicitly or implicitly examine program quality: 

♦ Local program monitoring or review conducted by the SEA to assess compliance 
or promote program improvement; 

♦ Criteria used by SEAs to evaluate local program funding applications; 

♦ Program evaluations of state or local adult education programs and evaluations of 
projects designed to improve program quality; and 

♦ Explicit indicators of local program quality already in use by states. 
Methodology 

To conduct the review, Pelavin Associates obtained copies of state monitoring and 
evaluation instruments from the state or from the state's most recent plan for adult education 
submitted to OVAE. After reviewing these documents, we contacted by telephone each state 
director of adult education to obtain additional published information about criteria for funding 
and on the specific quality indicators the state uses to assess local programs. A total of 46 
monitoring instruments and 20 proposal review protocols, requests for proposals (RFPs) or 
funding criteria were reviewed. 

To identify program evaluations, we conducted a comprehensive review of evaluations 
conducted since 1984. For evaluations to be included in this review, the study had to examine the 
impact of adult education instruction, staff development or other components of adult education 
programs and include indicators of program quality. Through data base searches, state directors 
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of adult education and reference lists from a previous iiterature review Pclavin Associates 
conducted in Fall, 1991, we identified 27 evaluations for review. 

To conduct the review, we first needed a definition of a "quality indicator." While there 
exists no standard definition, we defined indicators broadly as data or variables that represent 
program success in terms of efficient and effective operation. After review of several documents, 
however, we discovered that wide variation existed in the measures used to operationally define 
many indicators. For example, while there was wide agreement that the existence of a staff 
development component was an important indicator of program quality, there was general lack of 
agreement on how to define and measure quality staff development. Measures used included the 
existence of an inservice training requirement, hours of staff development time per staff member 
and staff input into staff development activities. Since we had no objective way to select which of 
these measures were most important, we included both the general indicator and the specific 
measures in our review. 

Review of Material 

Pelavin Associates abstracted the indicators of program quality within each monitoring 
instrument, funding protocol, and evaluation report. Each document was first reviewed to 
produce a list of the indicators and corresponding measures in the document. The indicators and 
measures were then organized by topic area within one of three component areas specified by 
OVAE: 

♦ Program context — needs, organization and structure of the program and 
characteristics of participants; 

♦ Program process and content planning, curriculum and instruction, qualifications 
of staff, and materials and equipment; and 

♦ Student outcomes — retention, educational, personal and social gains and work 
experience. 

Reviewers also maintained a tally of the number of documents that used each indicator. 
This procedure allowed us to summarize all of the indicators used, as well as to identify the most 
commonly used indicators for each type of document reviewed. 

Organization of Report 

The remainder of this report presents the quality indicators states use in determining local 
funding and in conducting program monitoring and evaluation. It also includes a section on 
explicit measures of quality used by states. Findings from each area are presented in separate 
sections. Section II presents monitoring indicators, Section III evaluation indicators and Section 
IV funding indicators. In these sections we first describe the documents reviewed and the 
methodology we used to obtain and review the documents. We next summarize the methodology 
the states or researchers use to obtain the information, followed by a summary of how the 
information is used. We then present a summary of the general topics and quality indicators used 
in a majority of documents under review. This information is presented in tabular form by 
component area specified by OVAE (context, process or outcomes) topic area, indicator, and 
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specific measure. The sections conclude with a detailed summary listing of all indicators and 
measures used by the documents, organized by OVAE's component areas. 



Section V of the report describes explicit quality indicators used in seven states. We 
present a description of the indicators by state, how the state measures and uses them, and how 
these indicators are tied to local program funding. 

In Section VI of this report, we present a list of quality indicators synthesized from all 
four sources. These indicators consistently appear in all attempts to measure the quality of adult 
education programs and thus represent current state consensus on indicators. 

Appendix A presents individual summaries of the program evaluation and Appendix B 
provides state summaries of funding criteria. 
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II. QUALITY INDICATORS IN STATE PROGRAM 
MONITORING INSTRUMENTS 



In all states, SEA staff visit local programs annually to assess program activities and 
operations. States refer to these visits as "monitoring" or "evaluation" of local programs and 
consider this activity a method of promoting program improvement. To conduct the review, states 
use a formal evaluation or monitoring instrument that specifies topic areas to consider, usually 
with specific sub-topics or indicators within each area. Pelavin Associates requested copies of the 
monitoring instruments currently in use from all states and received instruments from 46 states. 
In addition, we obtained further information on how monitoring visits were conducted and how 
the state used findings from the visit from the state director of adult education. 

Methods of Conducting Monitoring 

While a few states visit all programs at least annually, the majority of states visit from one- 
fifth to one-third of local programs each year. A single individual conducts the visit in many 
states, while some states use a team of evaluators that may include staff from other local programs 
or states, in addition to SEA staff. In some states, the local program staff completes a self- 
evaluation prior to the visit and the results of the two evaluations are compared. In other states, 
a short compliance visit is conducted annually and a more detailed visit is conducted every few 
years. The site visit lasts one or two days in most states, but is as long as a week in one state. In 
a few states, the review includes interviews with students and/or teachers or is supplemented with 
questionnaire data from students. 

Development and format of state monitoring instruments . States developed the topic 
areas covered in their monitoring instruments through adaptation from four sources: Federal or 
state legislative requirements, state standards for K-12 programs, the Federal standards for 
program quality developed in 1983 by OVAE, and accepted practice and research. The 
monitoring instruments employ one of the following five formats to record information from local 
programs. 

♦ Open-ended questions . The form lists topic areas or questions that the reviewer 
answers with written, open-ended narrative. Guidance on what to look for when 
addressing the topic is often included with the form. For example, one state's 
form includes the item "Staff development and network activities are carried out as 
described in the project application" and instructs the reviewer to look for whether 
staff development is available as described in the proposal. The reviewer 
responds to this item with a narrative on the program's staff development 
activities. 

♦ Yes/No response. This type of monitoring instrument lists the topical area and the 
reviewer checks yes or no to indicate its presence or absence. For example, "/ie 
there explicit learning outcomes for each student?" and "Are instructional methods 
tied to student's learning outcomes?" are answered this way on one state's form. 
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• Scales . The instrument uses three-, four- or five-point scales for each item to 
denote excellent, good, fair and poor or similar categories. Reviewers in one state, 
for example, respond to the item "Instructional contact hours are scheduled to fit 
student needs" by marking a single number, 1 - 5, to indicate "need major 
improvement" (1) to "excellent" (5). A few states use this approach without 
quantifying the categories. 

• Compliance with standards . In this approach specific standards are listed on the 
form, often along with indicators of the standard. The reviewer records whether 
the standard was met in some states. In other states, reviewers record "exceeds 
standard," or "needs improvement". For example, in one state the reviewer checks 
whether or not the program complied with the standard, "Wherever possible, the 
agency shall coordinate its adult education program with other basic skills activities 
in the same region." In a second state, whether the program meets, exceeds or 
needs improvement with the standard, "Specific job descriptions have been 
provided to teachers and staff," is recorded on the monitoring instrument. 

• Combinations . Several states combine two or more of these formats in their 
monitoring instruments. For example, one state uses the yes/no, scales and 
compliance formats. 

Exhibit 1 presents a summary of the monitoring instrument formats and the number of states 
using each format. 

Use of Monitoring Information 

States treat monitoring visits like individual case studies of how local programs are 
operating. Information obtained by the monitoring instruments is not easily quantified or 
aggregated across programs and few states do so. In addition, many of the standards and ratings 
on the monitoring forms are not objectively defined and require considerable judgment from the 
reviewer. 

The main purpose of program monitoring is to promote program improvement and 
identify technical assistance needs. The findings from the visit are also tied loosely to re-funding 
of local programs in most states. Following the visit the state prepares a report of the strengths 
and weaknesses of the program. The local program must address areas of concern within a 
certain time period or when re-applying for project funds. State directors reported that programs 
that did not satisfactorily address problems could be denied additional funds. This issue arises 
infrequently, however, since states work closely with local programs to correct problems and the 
local programs normally respond quickly to state concerns. In a few states, funding decisions are 
independent of program monitoring outcomes. 

Summary of Indicators on State Monitoring Forms 

While there were differences among states in the format of their monitoring instruments, 
there was greater consistency in what they measured and examined as part of monitoring. While 
there were some differences in organization and specificity of measures, the broad topical areas 
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EXHIBIT 1 



State Monitoring Instrument Formats 



Format 


iNUTODcr oi tsuiies using rormui 


Open-ended Questions 

Reviewer writes narrative description 
ot observations 


c 

J 


Yes/No Response 

Reviewer indicates presence or 
absence of indicator 


12 


Scale 

Degree of meeting standard indicated 
on three,- four, - or five-point scale 


8 


Compliance with Standard 

Reviewer indicates whether program 
met, exceeded, or needs to improve on 
standard or indicator 


13 


Combination 

Two or more of the above type of 
measures are used 


8 
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and measures of program quality commonly reflected in the monitoring instruments fall into five 
areas: 

♦ Planning (process, goals and relationship to program components) 

♦ Personnel Characteristics 

♦ Program Capacity (type of programming, instructional and other service providers, 
sites, and program management) 

♦ Curriculum/Instructional Methodology 

♦ Services to the Disadvantaged 

As these topic areas lend themselves to easy reclassification within the three program 
areas specified by OVAE for quality indicators - program context, process and outcomes - the 
following summary is organized around these topics. Exhibit 2 summarizes the topic areas 
addressed in monitoring the instruments of at least half (23) of the states reviewed. The summary 
below provides greater detail on specific indicators and includes additional indicators that a 
significant minority of states (15 - 22) deemed important. For each topic area the general 
indicator is listed followed by measures used on the monitoring forms. 

PROGRAM CONTEXT 

Need for Program Services 

♦ Program used a needs assessments to determine need for program services: 

a) availability and use of community-wide demographic data 

b) information on school and program dropouts, other adult education 
programs and job training program information 

c) annual documentation of needs 

d) services geared to reflect needs 

Organization/Structure of the Program 

♦ The locations of classes and nature of facilities are adequate: 



a) 


accessibility for handicapped 


b) 


accessibility to target population 


c) 


public transportation availability 


d) 


safety and comfort factors (cleanliness, lighting, etc.) 


e) 


spaces for storage, studying, intake and counseling 


0 


types of programs offered (ABE/GED/employment related/ESL) 


g) 


location - urban, rural, suburban 


h) 


intake procedures 
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availability of support services such as child care, transportation, 
and (asked by more than one-half of the instruments) availability 
counseling services 
referrals to other agencies 
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• Description of the organizational structure of the local delivery system: 

a) formal organizational charts 

b) communication and hierarchy within district 

c) job descriptions and clerical support 

• A minority of states asked about program scheduling: 

a) times of classes 

b) appropriateness of schedule 

c) flexibility of schedule 

Participants 

• Program enrollment levels and participation: 

a) enrollment numbers 

b) length of participation 

c) numbers by sub-population 

d) characteristics of the population served including demographic data, 
employment status, handicapped status, immigrants and numbers of 
high school dropouts 

• Enrollment numbers were compared to project goals, state rates and 
previous years' figures 

• A minority of states recorded whether the program had student objectives/ 
individualized learning plans 



PROGRAM PROCESS AND CONTENT 
Program Planning and Content 

• The program used an advisory committee to assist in planning: 



a) 


list of members 


b) 


target groups included 


c) 


broad based community representation 


d) 


ongoing input 


e) 


regular meetings; minutes available 


0 


addresses future needs 


g) 


has an action plan 


h) 


students involved in planning and goal setting 
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The program had goals and a plan that are: 

a) consistent with needs assessment 

b) consistent with program operations 

c) consistent with state and federal plan 

d) clearly written measurable objectives 

e) reviewed with staff, students, administrators 

f) annually revised 

The program coordinates with other related programs in the community: 

a) program has a coordinating committee 

b) coordination of services with employment and training programs, 
vocational/college programs, transportation and child care services, 
business, etc. 

c) use of volunteers 

Program has appropriate recruitment methods: 

a) student feedback on recruitment methods 

b) documented plan to recruit those most in need, evaluated annually 

c) encouraging non-traditional enrollment 

d) analysis of student data to determine if target population is being 
recruited 

e) use of multiple methods 

f) degree of satisfaction with community awareness, acceptance and 
support 

g) co-sponsored classes to facilitate recruitment 

h) working with other organizations to recruit 

i) who is involved in process 

j) evidence of materials to support recruitment efforts 
k) effort to follow-up with dropouts 

The program has an evaluation component: 

a) existence of an evaluation plan 

b) is a program evaluation done 

c) who is involved (faculty, students, staff) 

d) what is evaluated: staff, materials and methods 

e) how often is the evaluation done 

The program's record keeping is adequate: 

a) records properly kept 

b) types of records available 
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icutum, Instructional Materials and Equipment 



♦ G)urse content is organized and appropriate to meet learners* needs: 



a) 


availability of basic skills (pre-literacy and literacy programs) 


b) 


availability of life-coping skills program 


c) 


GED/ESL 


d) 


Career and employability instruction 


e) 


adult-oriented 


0 


integration of community resources and assessment 


g) 


content organized and sequential 




specialized short courses, accelerated programs 



• Description of the methods of instruction used: 

a) group, volunteer tutors, one-on-one, individualized, computer 
assisted, self-instruction, small group discussion, lectures, 
cooperative learning 

b) instruction shows evidence of planning 

c) variety of methods used/appropriate to learners 

• Methods of instruction for disadvantaged learners: 

a) course of study appropriate 

b) special materials for disabled students 

c) materials appropriate for special populations (ESL, disabled) 

d) listing of materials by type, level and grouping 

• Materials and equipment are adequate: 



a) 


variety and levels available 


b) 


A/V, computers available 


c) 


reflects occupational knowledge 


d) 


supports functional literacy 


e) 


appropriate for adults 


0 


sufficiency of material 


g) 


relevant and up-to-date 


h) 
i) 


free of bias, multi-cultural 


who selects materials 



• A minority of states asked additional questions relating to 
materials/equipment: 

a) use of supplementary resources 

b) instructor familiarity with materials 

c) access to library and library materials 

d) resources for outreach projects 

e) career guidance resources 
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Qualifications and Responsibilities of Personnel 

• Program assigns appropriate duties and responsibilities to the progrn 
director/administrator: 

a) program management 

b) personnel 

c) reporting 

• Staff understand their jobs and role in the program: 

a) staff have job descriptions 

b) regularly scheduled staff meetings 

• Staff understand and remain knowledgeable of the program: 

a) orientation for new staff 

b) regularly scheduled staff meetings 

• Staff are qualified and have appropriate experience: 

a) teachers with certification 

b) teachers working toward certification 

c) training of teachers 

d) prior experience in adult education 

e) certification in elementary/secondary education 

f) bachelor's degree 

g) membership in adult education profession society 

h) educational level of aides 

i) qualified staff required 

j) appropriate salaries provided 

• Program has a staff development component: 

a) evidence of availability of staff development activities 

b) content of staff development activities 

c) input to staff development plan 

d) effectiveness of staff development activities 

e) goals and objectives of staff development plans 

f) evaluation of staff development activity 

g) ongoing staff meetings with updates of trends, issues 

h) opportunities for professional growth 

i) Opportunities for off campus staff development 
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♦ A minority of states asked about additional knowledge and characteristics 



of staff: 


a) 


creativity/communication skills 


b) 


interesting and knowledgeable 


0 


good morale and enthusiasm 


d) 


knowledge of content area 


e) 


knowledge of support services 


0 


use of lesson plans 


g) 


address individual needs 



PROGRAM OUTCOMES 
Retention 

♦ A minority of states ask about student follow-up: 

a) evidence of provision for follow-up of leavers 

b) evidence of provision for follow-up of completers 

c) data used in program reviews 

d) contacting of frequently absent students 

e) identification of method of contact 

Educational Gains 

♦ Assessment and placement of students: 

a) availability of diagnostic files 

b) appropriateness of methods of student assessment 

c) placement methods at enrollment 

d) use of tests for GED and ESL students 

e) factors to be considered for placement 

♦ Student progress and learning gains: 

a) progress reports including records kept, use of individual folders, 
use of records by students/staff 

b) availability of testing, counseling, evaluations, diagnostic assessment 

c) instruments used in assessment 

d) student progress toward goals 

e ) progress testing occurring on regular basis 

f) test results and scores 



17 



ERLC 



25 



HI. QUALITY INDICATORS FOUND IN PROGRAM EVALUATIONS 
CONDUCTED FOR ADULT EDUCATION PROGRAMS 



Pclavin Associates conducted a comprehensive review of program evaluations of adult 
education programs to identify quality indicators researchers believed to be important in assessing 
program effectiveness. In this review we included studies that have been conducted since 1984 
that have examined the impact of adult education instruction, staff development or other 
component of adult education programs; and that include indicators of program quality. Studies 
were identified through data base searches, state directors of adult education and reference lists 
from a previous literature review Pelavin Associates conducted in Fall 1991. 

Our review identified 27 studies that evaluated or assessed evaluations of state programs 
overall, local programs, literacy projects, staff development projects, volunteer programs and 
workplace literacy programs. Evaluations of ABE, ESL, ASE and GED studies were included. 
Exhibit 3 presents a list of the studies included in our review. Topics evaluated in these 
documents included statewide evaluations of AE programs in terms of structure and function; 
follow-up surveys of students; promising practices in effective programming for adult education; 
state plan implementation strategies and the results of a learner outcome study of community- 
based literacy programs. 

Methods of Conducting Evaluation 

Most evaluation activities were conducted by the agency administering the program (SEAs 
or community colleges) although several states contracted with local universities or consulting 
firms for the evaluation activity. When the evaluation was carried out by the administering 
agency, state staff as well as peer administrators and teachers representing other local programs 
were involved. Most studies involved interviews with project staff and students. Follow-up 
studies of students included test scores, goal attainment and other outcome measures. In contrast 
with monitoring activities, most evaluations are not conducted annually. 

Project Advisory Committees, project staff, state and local program directors, 
administrators and, in at least one instance, cooperating agencies provided input to evaluators on 
the selection of measures used in the evaluation. 

The main purpose of most of the evaluations was to determine how effective the program 
was in following generally accepted strategies for good programming and determining the impact 
of the program on its participants. 

Summary of Indicators Found in Adult Education Evaluation Activities 

Indicators used in evaluations varied widely. This variation results from the differing 
purposes of the evaluations, as well as the priorities various administering agencies place on these 
elements. 
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EXHIBIT 3 



Evaluation Studies Reviewed 



Alamprese, J., Keltner, A, & Savage, KJL (1988). The ESL Teacher Institute: Its 
Impact and Future . Sacramento, CA: Association of California School 
Administrators, Foundation for Educational Administration (ED 323 756). 

Bonnet, D.G. (April 1988). "An Evaluation of Adult Literacy Efforts in Indiana." 

Submitted to the Indiana Adult Literacy Coalition. Indianapolis: D. Bonnet 
Associates. 

Brewington, L.A (1985). "Program Evaluation and Implementation Report. Adult Basic 
Education Program/ Raleigh: North Carolina Commission of Indian Affairs (ED 
267 938). 

Evaluation and Training Institute. (1990). Evaluation of the Federal Four- Year Plan for 
Adult Education. First-Year Interim Report. Submitted to Adult Education Unit, 
California Department of Education. Los Angeles. 

Fellenz, R.A, & Conti, G.J. (1984). Comprehensive Evaluation of the Statewide Texas 
Adult Basic Education Program: Evaluating a Human Enterprise . Prepared for 
the Texas Education Agency, Division of Adult and Community Education. 
College Station, TX: Texas A&M University (ED 260 213). 

Fingeret, A, et al. (1985). North Carolina Adult Basic Education Instructional Program 
Evaluation, 1985. Raleigh: North Carolina State Department of Community 
Colleges (ED 269 622). 

Fingeret, H.A, & Danim, S.T. (1991). "They Really Put a Hurtin' on My Brain': 
Learning in Literacy Volunteers of New York City." Executive Summary. 
Durham, NC: Literacy South. 

Formative Evaluation Research Associates. (1990, August). "Evaluation Proposal. 
Michigan Adult Literacy Initiative Project." Ann Arbor, MI. 

Formative Evaluation Research Associates. (1990, August). "Evaluation Proposal. Staff 
Development Collaborative Project." Ann Arbor, ML 

Georgia Department of Technical and Adult Education, Office of Adult Literacy 
Programs. (1991). A Follow-Up Study of Graduates in Georgia . Atlanta. 
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EXHIBIT 3 (Continued) 
Evaluation Studies Reviewed 



Goodman, R.L. (1989). A Final Report. July 1, 1988 to June 30, 1989. Project Quality . 
For the Conduct of Adult Education Program Evaluations in Illinois. Prepared in 
cooperation with the Illinois State Board of Education, Adult Education Section 
and Springfield Public School District No. 186, Lawrence Adult Education Center, 
Springfield, IL: Adult and Continuing Education Consultants, Inc. 

Havlicek, L.L. (1990), "External Evaluation of the Kansas Adult Education Program for 
Fiscal Year 1990." Lawrence: University of Kansas. 

Haley, WJ., & Hoover, W. (1985). Final Report. Follow Up and Impact Studv of Adult 
Basic and Secondary Education Students . Harris County, TX: Harris County 
Department of Education and Southwest Educational Laboratory. 

Hughes, D.M., & Brannon, Y.S. (1988). An Evaluation of Adult Basic Education 
Programs in North Carolina . Raleigh: Center For Urban Affairs and 
Community Services, North Carolina State University. 

Iowa Department of Education. (1990). Assessment and Adult Basic Education: The 
Iowa Model. Des Moines. 



Jackson, G. (1989). ACBE Evaluations of Community Based Literacy Programs H988- 
89) . Washington, D.C.: Association for Community Based Education. 

Louisiana Department of Education, Bureau of Adult and Community Education. (1990). 
"Evaluation Report for 1989-90." Baton Rouge. 

Mahaffy, J.E. (1990). Final Report: Adult Basic Education Student Follow-up 

Evaluation . Submitted to Adult Education and Literacy Programs, Office of the 
State Superintendent of Public Instruction, Olympia, Washington. 

Mahaffy, J.E, (1983). "Impact Evaluation of Adult Basic Education Program Outcomes." 
Helena: Montana State Office of Public Instruction, Division of Adult Education. 

Mikulecky, L., & D'Adamo-Weinstein, L. (1990). "Workplace Literacy Program 

Evaluations." Paper presented to the Work in America Institute, Harvard Club, 
November 7, 1990. 

National Adult Literacy Project. (1984). Guidebook for Effective Literacy Practice, 1983- 
1984 . The Network, Inc., and Far West Laboratory. 
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EXHIBIT 3 (Continued) 
Evaluation Studies Reviewed 



Nelson, D., & Bringewatt, M. (1986). Report on the Evaluation of Adult Refugee ESL 
Programs in Minnesota . St. Paul: Minnesota Department of Education, Office of 
Community and Adult Basic Education. 

Sherman, J.D., Kutner, M.A., Lancaster, M., Suitor, J.H., & Tan, A. (1991). Evaluation of 
the Virgin Islands Adult Education Program . Submitted to the Commissioner of 
Education, Virgin Islands Department of Education. Washington, D.C.: Pelavin 
Associates, Inc. 

Sherron, R.H. (1986). A Longitudinal Evaluation of Adult Basic Education in Virginia: 
19704985 . Richmond: Virginia Department of Education (ED 276 853). 

Smith-Burke, M.T., Parker, M., & Deegan, D. (1987). "Starting Over: Characteristics of 
Adult Literacy Learners." New York: Literacy Assistance Center. 

Snow, E., & Bentley, N. (1988). Adult Education in Texas Program Evaluation FY 1986- 
88. Austin: Texas Education Agency. 

Sticht, T.G. (1991). "Evaluating National Workplace Literacy Programs." El Cajon, CA: 
Applied Behavioral & Cognitive Sciences, Inc. 
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Our review identified over 60 measures of program quality, with some measures used in as 
many as 12 studies. Despite the wide variation, we identified four topics that were included in 
more than half of the evaluations: 



• Characteristics of program participants; 

• Community representation and involvement in program planning; 

• Recruitment strategies; and 

• Participant outcome measures. 

The indicators were reclassified within the three program areas of program context, 
content and outcomes. Exhibit 4 summarizes the indicators used in at least five of the evaluation 
studies. The remainder of this section presents individual summaries of each study reviewed, 
providing a brief overview, indicators used and how the indicators were selected. 
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IV. QUALITY INDICATORS IN STATE FUNDING CRITERIA 



As part of our discussions with state directors of adult education on quality measures, we 
asked each director to describe his or her state's criteria for funding local programs and to 
provide us with a copy of protocols used by raters to evaluate proposals. We received the state 
rating protocol, application guidelines or request for proposals (RFP) from 15 states and obtained 
detailed information verbally from an additional five states. 

Eleven of the states had formal review protocols with numeric rating scales. In these 
states, a review team assigns a numeric score to proposals and determines a cutoff point for 
funding. The remaining nine states reviewed funding requests using a list of topics and general 
guidelines for review. In addition to formal criteria, several states gave priority to programs that 
had received funding in the previous year and had performed satisfactorily. Most state directors 
reported that their state would be revising funding procedures in response to new requirements of 
the National Literacy Act. 

Summary of Quality Indicators on State Funding Protocols 

The criteria on state funding protocols can be considered indicators of program quality 
which states believe to be important for successful program operation. Pelavin Associates 
reviewed information from each of the 20 responding states and synthesized the topic areas, 
indicators and measures. The indicators were organized around two of the three topic areas 
(program context and process) specified by OVAE. State funding protocols did not include 
student outcome data. 

From our review, it is clear that states evaluate proposals from local programs according 
to two main criteria: (1) good attention to good planning and program design in general, not 
necessarily specific to the Adult Education Act; and (2) in some instances, reflection of certain 
characteristics unique to the Adult Education Act, such as special services to lowest education 
levels, and target populations like the homeless and handicapped. The main topics addressed by 
the protocols include: 

• Needs assessment 

• Participants served, including target populations 

• Program goals and objectives 

• Location of classes 

• Curriculum and instructional program 

• Staff qualifications 

• Coordination with related programs 

• Evaluation plan 
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Our analysis identified 13 topics and indicators examined by state funding protocols. 
Exhibit 5 presents the indicators and measures found in al least one third (7) of the protocols or 
criteria. A summary of all indicators and measures used is listed below. 

Program Context 

• Documentation of need for services 

Data on needs of population in service area (through needs assessment 
and/or demographic data) 

• Targeting of services to special populations 

Specific populations targeted 

Services targeted to handicapped or other specific population 

• Number and types of participants 

Number of participants to be served 

Number of participants to be served by demographics and incoming skill 
levels 

• Location of facilities and scheduling of classes 

Flexible scheduling at different times and days 
Number of service sites 

Locations convenient to participants (in neighborhoods or readily accessible 
locations) 

Program Process and Content 

• Adequacy of facilities 

Facilities large enough and in good condition 
Facilities accessible to handicapped 

• Availability of support services 

Child care available 
Transportation available 
Counseling and referral available 

• Curriculum and quality of instruction 

Instruction related to student needs 
Instructional hours 

Student assessment to customize instruction 
Individual learning plans developed 
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Life skills curriculum used 
Structured instructional program 

♦ Coordination with related programs 

Involvement of other programs through referrals, joint planning, shared 
resources 

Funding from other programs 
Employability focus included 

♦ Clear planning with measurable goals and objectives 

Measurable, attainable goals specified 

Goals relate to state goals 

Goals reflect needs of population 

Goals reflect innovation and program expansion 

Previous goals and objectives met 

♦ Recruitment and outreach methods 

Program promotion and recruitment to target population 
Open entry/exit 

Ability to recruit the disadvantaged 
Met previous recruitment goals 

♦ Tracking of participants 

Accurate tracking and record keeping 
Records able to document program impact 

♦ Evaluation plan 

Workable evaluation plan specified 
Specific evaluation instrument specified 

♦ Qualifications of staff 

Evidence of staff qualifications (degree, certificate) 
Existence of staff development activities 
Volunteer staff used 

In addition to these categories, the appropriateness of the budget and amount of local and 
additional resources for the program were included in all funding criteria examined. Appendix B 
presents a state-by-state summary of funding criteria. 
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V. EXPLICIT STATE QUALITY INDICATORS 



Seven states have developed additional indicators of program quality beyond monitoring 
and funding requirements. These indicators explicitly define for local programs criteria or 
outcomes the SEA will use to assess local programs. Programs are held accountable for their 
performance on the indicators and in three states, program funding is - or will be - tied directly 
to the indicators. In five states, the indicators address student outcomes while indicators in two 
states focus on program content. Five of the states have - or will develop - quantitative 
measures of the indicators and one of these states (Massachusetts) is developing performance 
standards to judge local program performance. In the remaining two states (California and 
Michigan) the indicators are described qualitatively. Exhibit 6 summarizes the indicators for each 
state. Exhibit 7 summarizes the quantitative indicators for the five slates that have them. The 
indicators used are described in greater detail for each state below. 

ARKANSAS 

In 1988 the Vocational and Technical Education Division of the SEA developed 
Effectiveness and Efficiency (E & E) criteria for adult education programs in response to the 
state board of education's concerns to improve program quality. The criteria measure enrollment, 
cost, use of staff and student outcomes. To be considered for funding, a program must meet the 
criteria and be certified "Effective and Efficient." The state makes this determination of all 
programs each March. Using the data the state maintains on computer for all programs, state 
staff assign a numeric score to each of the seven E & E criteria: 

♦ Percent of eligible students served compared to the statewide rate; 

♦ Percent eligible students enrolled compared to the statewide rate; 

♦ Average cost per student instructional hour compared to the state average; 

♦ Reasonable maintenance and operation costs; 

♦ Efficient use of full-time staff; 

♦ Percent of students attending 40 or more hours who advance (measured through 
grade level equivalent); and 

♦ Number of students who pass GED Tests compared to the number who take the 
tests. 

Each criteria is assigned from 10 to 16 points for a total score of 100. Programs must 
attain a minimum score of 75 to achieve E & E status. Specific scoring guidelines are provided 
for each item. For example, a program receives 12 points if the percent of eligible students 
enrolled equals 50 to 74 percent of the state rate and the maximum points, 16, if the percentage 
enrolled equals 150 percent or more of the state rate. 
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A program that docs not attain E & E status must initiate corrective action. The state 
notifies the program and arranges a meeting with the local administrator to explain deficiencies. 
The local program must submit a written improvement plan within one quarter that delineates 
improvement activities, goals and a timeline for improvement that does not exceed one year. The 
local education agency monitors improvement quarterly and if the program fails to achieve E & E 
status within one year, a second improvement plan must be developed. If E & E status is not 
achieved after the second year, the state may terminate funding of the program. To dale, no 
program has been denied funding, although several programs have now failed to achieve E & E 
status for the third year and may be terminated this year. 

The state director reports that only smaller, part-time programs have had serious difficulty 
achieving E & E status. He reported considerable support within the state for the criteria as a 
means of improving program quality. 

CALIFORNIA 

In 1988 the California Superintendent of Public Instruction established an adult education 
advisory committee to guide educational reform in the state's adult education program. The 
committee prepared a strategic plan that included recommendations to develop quality standards 
and performance measures to guide program development and evaluation. In response to the 
report, the Adult Education Unit of the State Department of Education has begun the process of 
developing standards and measures for ABE, ESL and ASE programs. The standards will be used 
(1) to identify items that can be used in checklists for program monitoring, (2) to guide 
development of inservice programs and teacher evaluation, and (3) to identify student outcomes 
to assess student achievement. The indicators will not be tied directly to local program funding, 
but compliance will be verified through program monitoring. 

The state has recently completed development of standards for ESL programs with the 
assistance of a field-based ESL curriculum committee. The state's draft document Model 
Standards for Adult English-as-a-Second Language Programs divides the standards into three 
areas: general standards, which include program, curriculum, instruction and evaluation; 
proficiency levels, with seven levels defined; and ESL testing, which describes standards for ESL 
testing. Specific measures are not identified for each standard, but representative examples are 
presented to illustrate how the standard could be implemented and assessed. The model 
standards are summarized below. 

General Standards 

• Program Standards - programs should have the following components: 

Articulated sequence of ESL courses 

Curriculum that identifies objectives for each course 

System to assess language proficiency 

• Curriculum Standards — the curriculum should have the following characteristics: 

Targeted to meet student needs 
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Teaches English by integrating vocabulary, grammatical structures and 
language functions 

Considers student's lack of literacy ability 

• Instruction - instructional activities should have the following characteristics: 

Integrate the four language skills (listening, speaking, reading and writing) 

Include language tasks that consist of meaningful interchanges 

Focus on acquisition of communication skills 

Focus on development of receptive skills prior to productive skills 

Use different grouping strategies to facilitate student-centered instruction 

Be varied to address different learning styles 

Integrate language and culture 

Allow development of language necessary to express higher thought 
Integrate language and critical thinking 

• Evaluation — the program's evaluation component characteristics should include: 

Placement into course level determined through variety of assessments 

Monitoring of student progress on a continual basis 

Assessment for exiting levels includes proficiency and achievement testing 

Proficiency Levels 

• Seven language proficiency levels are defined: Beginning literacy, low beginning, 
high beginning, low intermediate, high intermediate, low advanced and high 
advanced. For each proficiency level there is a description of student skills and 
course content. 

• Course content describes what students will be able to do on completion for the 
four major language skills and identifies language functions and language forms for 
each level. 

Testing Standards 

• Program testing of students should be characterized by: 

Use of valid and reliable tests developed for non-native English speakers 

Interpretation based on adult ESL proficiency levels 

Use of tests to gain information that benefits students 

Providing students with multiple opportunities to demonstrate their abilities 

in more than one area 

Determination of proficiency through a variety of assessment measures 
Content that incorporates everyday, real-life situations and materials 
Allowing application of higher order thinking skills 
Administration to ensure fair and accurate results 
Standardized scoring procedures 
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ESL programs will begin using the standards within the next few years. The state is 
currently working on standards for ABE and ASE programs. 

MASSACHUSETTS 

Massachusetts has been planning and field testing its ABE Accountability System over the 
past four years and will be continuing development through June 1993. The process will 
culminate in a set of performance standards for ABE programs that assess (1) program focus or 
inputs and (2) learner focus or outcomes. The standards, to be implemented beginning in fiscal 
year 1994, will be set separately for different target populations and will be tied to local program 
funding. 

The program focus or input standards will measure whether local programs are successful 
in translating their program goals into curriculum and instructional programs. These standards 
will be progress markers of the overall program. The learner focus or outcome standards will 
assess three areas: 

♦ Learner participation , such as enrollment levels and attendance; 

♦ Learner goal attainme nt - whether the learner achieves the stated goal for ABE 
participation; and 

♦ Benchmark measures toward goal attainment, including skills and competencies 
needed to attain the goal The state is considering using CASAS to develop 
benchmarks and is also reviewing existing instructional materials to determine 
whether benchmarks may be developed from them. 

Specific indicators and standards have not yet been selected, but will be finalized after a 
16-month period of field tests has been completed. There are 10 components to the field tests, 
organized within three tiers as described below. 

♦ Tier 1 . The purpose of Tier 1 is to provide information for the development of 
specific indicators and performance standard levels. There are four component 
studies to this tier, each of which will be conducted through a pilot test involving 
10 local programs: (1) a pilot test of protocols developed earlier that will measure 
participation, goal attainment and benchmarks for different learner populations; 
(2) a study of how local programs develop program goals, translate these goals into 
instruction and measure the success in meeting their goals; (3) a study to examine 
alternative assessment methods, such as portfolio assessment, and examine how this 
type of assessment information could be adapted to meet state reporting needs; 
and (4) an examination of how other education programs (not funded through the 
state ED or by JTPA) develop and measure program and learner goals. 

♦ Tier 2 . Tier 2 will study existing state reporting systems to determine how well 
they match data requirements that will arise from Tier 1. There are three 
component studies comprising this tier. These studies will examine: (1) the utility 
of existing evaluation instruments used by the state ED to collect data; 10 
programs will be examined to make this analysis; (2) a survey of teachers will be 
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conducted to assess the effectiveness of using elements developed by the state's 
System of Adult Basic Education Support (SABES) to assess program 
effectiveness; and (3) a survey of 250 adult learners will identify factors learners 
believe are important in assessing their goal attainment. 

• Tier 3 . The third tier assesses the state's own accountability in achieving its goals. 
The three component studies of this tier include a self evaluation of the state ED 
to determine how well it supports the development of a diverse network of high 
quality ABE programs, a review of evaluation systems used in other countries and 
a state "report card" of progress made in developing an ABE accountability system. 
The state ED will survey the opinions of executives and heads of relevant state 
agencies to make this assessment. 

Local programs that will participate in the field tests will receive substantial additional 
funding to conduct the study. The state is currently in the process of selecting field sites. 

MICHIGAN 

Michigan's quality standards serve as the basis for its program monitoring. However, the 
state differs from other states by requiring programs to address the standards explicitly in two 
ways. First, local education agencies must certify to the state annually that the local plan for adult 
education incorporates the quality standards. Second, each program must submit an annual report 
of how it is complying with the state standards. The program develops the report after 
conducting a self-evaluation using a standard form, which has items that reflect the quality 
standards. As part of its on-site monitoring visit, the state review team compares the information 
in the compliance report with its own observations. The 10 quality standards do not specify 
quantitative measures but are stated narratively as summarized below: 

• Institutional purpose — explicit program goals, objectives and mission statement; 

• Organization, administration and control - professional staff and efficient 
administration; 

• Instructional program — designed to meet student needs and goals, including 
employment needs; 

• Professional staff — including quality teachers and other staff; 

• Student personnel services — including assessment and counseling; 

• Institutional adaptability - the program is able to change, adapt and innovate to be 
more responsive to its students; 

• Instructional media program - provides a wide range of media for staff and 
students; 

• Financial support and control « the program has sufficient funds and financial 
controls; 
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• Facilities, equipment and supplies « are of high quality and meet program and 
learner needs; and 

• Evaluation - the program uses evaluative information for program improvement. 

In addition, each program must have a plan for retention, follow-up and placement of 
students and report on its activities in these areas as part of the compliance report. Compliance 
to the standards is not lied directly to funding. However, the program must address resolution of 
any problems in its next funding application. 

Michigan's quality standards will be substantially revised this year in response to the state's 
Quality Incentives Act. This new law is aimed at promoting school improvement and requires K- 
12 programs, as well as adult education programs, to (1) develop a school improvement plan, (2) 
implement outcome-based curriculum, (3) use accredited facilities and (4) submit an annual report 
to the state on progress in implementing these areas. Programs have until September to develop 
a school improvement model. After that time, the SEA will reorganize the indicators around the 
topic areas addressed by the Act (school improvement, curriculum and accreditation) and develop 
new quality indicators where needed, such as to measure progress toward school improvement 
strategies and to incorporate the current retention, placement and follow-up requirements into a 
specific standard. 

TENNESSEE 

Tennessee has established a goal of achieving a 90 percent literacy rate by 2000. To help 
meet this challenge, the SEA in Tennessee has established three competency levels for ABE, 
defined loosely along grade levels (0 - 4, 5 - 8, 9 - 12). A set of proficiencies is established for 
each level as measured by the ABLE or Slosson tests. A student may enter ABE at any level and 
progresses to the next level after meeting the proficiencies defined for the level. Students receive 
a certificate after completing each level and progress until passing the GED Tests. They then may 
continue their education by moving into an ASE instructional program. 

Tennessee's delivery system is county-based, with each county having a local adult 
education director, advisory board and at least one fiscal agent. All ABE funding is awarded to 
the fiscal agent, which is then responsible for coordinating ABE programs in the county. To 
achieve the state literacy goal, the state establishes a fixed number of graduates annually for each 
county, based on the number of people in the county with less than a high school education. The 
ABE programs in the county must graduate at least 90 percent of this number of students. 
Beginning in program year 1993, funding to the county will be reduced if the county fails to 
achieve its graduation target. 

UTAH 

Utah's director of adult education developed a set of program quality indicators that were 
first implemented in 1990. All local programs must report to the state, a total of 32 quantitative 
measures in five topic areas through the statewide management information system. Programs 
maintain individual student records on this computer system, which facilitates the annual data 
collection. The list below summarizes the measures. 
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(1) Educational Effectiveness 



• Increased enrollment 

Number of students enrolled in each of four literacy levels 
Number of students passing GED, obtained GED certificates, 
graduating high school or entering another education and training 
program 

• Outcome achievement 

Average number of clock hours, per student 
Units of credit awarded 

(2) Target Populations 

• Increased enrollment of target populations 

Enrollment of: racial/ethnic educationally disadvantaged, rural and 
urban residents, limited English speakers, immigrants, correctional 
or institutionalized adults, older adults, homeless, handicapped, 
public housing residents 

(3) Economic Impact 

• Improved employment-related outcomes 

Number of enrollees who obtain a job, increase salary and removed 
from public assistance 

(4) Mastery Achievements 

• Increased outcome achievements 

Number of units of credit awarded to students by four skill levels 

(5) Social Impact 

• Increased personal achievements 

Number of students with enhanced self-confidence and personal 
satisfaction 

Number of students who leave after and before completing 
educational objectives 

Number of students used as volunteers in the program 
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Due Id the state's automated data base, programs have not had difficulty reporting these 
measures to the state. 

The state director reports that the most meaningful outcome measure is the unit of credit 
awarded. These credits are awarded to a student upon demonstrated competency in skills defined 
in the program's curriculum. Since the definition is objective, awarding of a credit represents 
attainment of an objectively defined skill. 

Local program performance on the indicators is not tied directly to funding but is 
evaluated as part of the state monitoring visit. If a program is performing poorly on any measure, 
the issue is discussed with program staff and corrective action planned. A pattern of poor 
performance could affect future funding if the cause was believed to be under the program's 
control. However, this problem has never occurred to date. 

VIRGINIA 

Virginia is in the process of developing quality indicators and is currently conducting a 
pilot project testing indicators of administration and instruction. After the pilot project is 
completed, the state plans to implement the indicators throughout the state over the next few- 
years. Three indicators have been defined: 

• Recruitment whether the program met its recruitment targets; 

• Staff development - a minimum of 10 hours of staff development activity for each 
staff member; and 

• Student goal attainment — whether students achieved their goals for attending the 
program. 

On intake into the program, the instructor determines the student's goals and then 
develops an instructional plan that will enable the student to achieve these goals. For example, if 
the student enrolled in the program to learn to help his or her child with homework, the 
instructor asks the student to bring samples of the homework to class. The teacher would then 
review the skills required to complete the homework and would design an instructional plan to 
teach the student these skills. These skills are then specified in an individual learning plan and 
the teacher determines the degree to which the goal is attained by the percentage of the skills the 
student learns. 

Similarly, if the student's goal is to get a job, the student selects the job and the teacher 
determines the skills the student would need to obtain it. The teacher then develops an 
instructional plan to teach these skills and again measures goal attainment by the percentage of 
skills learned. 

The instructor is responsible for determination of student goals, criteria for attainment and 
actual goal attainment. The determination is made on an individual basis and there are no 
statewide standards for determining attainment. 
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Virginia has no plans to tie these measures to program funding. The state director 
believes linking funding to the indicators may cause programs to selectively screen participants at 
enrollment and "cream" the more qualified students. In addition, programs may be more likely to 
provide inaccurate data if funding was at stake. To avoid these problems, no program will be 
penalized due to poor performance on the indicators, although the state would try to improve 
problems through consultation and assistance. 
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VI. Synthesis of Quality Indicators Used By SEAs and Program Evaluation 



In this section, we present a summary of quality indicators used by SEAs and in program 
evaluation. To compile this list, we synthesized the indicators presented in Exhibits 2, 4, 5, and 7. 
These indicators appear consistently in each of the three areas where states currently assess 
quality - local monitoring, program funding, and evaluation - and several of the indicators are 
already used explicitly by five states (see Exhibit 7). The list also includes measures of quality 
frequently used by outside researchers evaluating aspects of adult education programs. The list 
can thus be considered an empirically derived consensus of SEAs on appropriate quality indicators 
for adult education programs. 

The list organizes the indicators within the three component areas specified by OVAE. 
In each component area, the topic is first listed, followed by specific indicators for each topic. 
Within the program process and content area, indicators are divided into four topic areas: 
program planning, program content, staff qualifications and responsibilities, and curriculum and 
materials. 
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SAMPLE QUALITY INDICATORS 
FOR ADULT EDUCATION PROGRAMS 
USED BY SEAs AND IN PROGRAM EVALUATIONS 



PROGRAM CONTEXT 
Documented Need for Program Services 

A. Number and demographics of target populations in need 

B. Literacy levels in community 

C. High school drop-outs in community 

D. Employment-related skill needs of community 

Organization and Structure of Delivery System 

A. Number of projects 

B. Variety of locations and settings of projects 
C Varied types of projects (e.g. t ESL, GED) 
D. Flexible scheduling 

Characteristics of Participants 

A. Number and demographics of participants 

B. Number and demographics of participants by skill level 

C. Number and demographics of participants by program type 

D. Number and demographics of participants by program setting 
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PROGRAM PROCESS AND CONTENT 
I. PROGRAM PLANNING 



1. Community and Staff Input in Program Development 

A. Existence and use of an advisory board 

B. Program holds public hearings 
G Use of staff input 

D. Other sources consulted (e.g., employers, staff, CBOs, evaluations, program 
performance reviews) 

2. Coordination Activities 

A. Existence of coordination arrangements (formal or informal agreements, agencies 
involved — number and type) 

B. Type of coordinated activities: referrals; share staff and/or facilities; joint planning 
and budgeting 

3. Written Operational Plan 

A- Existence of a plan 

B. Measurable goals and objectives specified 

G Specific program goals and objectives specified consistent with state plan 
D. Plan development process includes broad input and is open to change 

II. PROGRAM CONTENT 

1. Recruitment 

A. Recruitment methods used 

B. Special populations targeted 

C Program outreach and publicity activities 
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2. Program Intake Procedures 

A. Entry policies 

B. Incoming assessment procedures 

C. Development of individual learning plans 

3. Ongoing Assessment Methods 

A. Procedures for monitoring student progress and learning gains 

B. Procedures for monitoring student progress toward goals 

4. Support Services 

A Type of support services offered (e.g., counseling, transportation, child care) 
B. Adequacy of services for meeting student needs 

5. Evaluation 

A. Student, community and staff evaluation of program activities 

III. STAFF QUALIFICATIONS AND RESPONSIBILITIES 

1. Characteristics of Staff 

A. Demographics 

B. Educational background, credentials, experience 

C. Number of staff 

D. Staff retention 

2. StafT Responsibilities 

A. Duties of staff appropriate for position 

B. Staff commitment (e.g., full or part-time, additional duties) 
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3. 



Staff Development 



A. Existence of staff development component: 

1. When provided (pre- or in-service) 

2. Content, topics covered 

3. Duration 

4. Sequential training 

5. Staff compensation for attendance 

B. Evaluation of staff development activities: 

1. Systematic needs assessment for content 

2. Evaluation of activities by staff 

3. Staff participation in development 

4. Use of Volunteer Staff 

A. Duties of volunteers 

B. Volunteer training 

5. Evaluation of Staff Performance 

A. Methods for evaluating staff 

IV. CURRICULUM AND MATERIALS 

1. Type of Curriculum and Instruction Used 

A- Instructional methods meet student needs (sequential, individual, competency- 
based) 

B. Topical emphasis relevant to adult learners 

C. Instructional technique (e.g, peer teaching, small group) 

D. Amount of instruction offered 

E. Organized sequence of courses used 

F. Individualized instruction based on assessment 
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2. Materials and Equipment Used 

A. Adequate materials used: 

1. Appropriate for student abilities 

2. Appropriate for student interests and needs 

3. Reflect diverse socioeconomic and cultural background of learners 

B. Adequate equipment used: 

1. Appropriate to meet program and learner needs 

2. Sufficient amount to meet program and learner needs 

3* Selection and Evaluation of Materials Equipment 

A. Method used to select and evaluate equipment and materials (e.g., instructor and 
student input) 

PROGRAM OUTCOMES 

1. Retention and Follow-up Methods 

A. Hours of instruction received 

B. Participation rates for population subgroups 

C. Methods for contacting program leavers 

D. Exit interviews conducted 

2. Educational Gains 

A. Grade level advancement 

B. Competencies attained 

C. GED or high school graduation attained 

3. Employment 

A. Attained new employment 

B. Improved current employment 
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C. Improved income 

D. Attained employment-related skills 

E. Receiving public assistance 

Personal and Social Goal Achievement 

A. Achieved personal goals for participation 

B. Improved self-esteem and self-confidence 
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The ESL Teacher Institute 
California 

KvalH tttion Overview 

This study provides an evaluation of the 1986-87 Institute, a staff development program 
for preparing beginning and experienced California instructors to use ESL techniques and 
materials. The purpose of the evaluation was to learn about the Institute's impact on 
(1) improving instructors' skills in utilizing ESL techniques; (2) instructors' capacity to use 
compctency-buseo .dull education (CBAE) classroom management methods; (3) instructors 1 
abilities to use sequential steps in specific ESL techniques (e.g., listening, speaking, and reading); 
and (-?;■ instructors 1 and trainers* perceptions about their own skill development through systematic 
training and reinforcement. 

Me asures of Prog ra m Quality 

Program Process and Content 

♦ Evidence that instructors demonstrate eight ESL techniques: role play, focused 
listening, early production, reading, drills, language generating, dialogues, and pair 
practice. Examples of teacher application of the techniques include*the following: 

-Use lesson plans to conduct lessons 

-Use competency objectives to focus lessons 

-Set up situation by asking questions, providing visuals, relating it to previously 

studied curriculum, eliciting ideas or experiences from students 

-Prepare students to understand the language they would hear 

-Provide tasks that require students to demonstrate comprehension; provide 

examples for tasks 

-Provide materials to guide students in listening (e.g., realia, visuals, print) 
-Present language with normal speed, intonation, stress and volume; provide 
comprehensible input through visuals, realia, gestures and body movement 
-Provide feedback to learners on the accuracy of their listening, pronunciation 
-Provide context for new vocabulary 

-Model expected dialogue/responses before asking students to respond 
-Move from whole group to small group or individual practice 
-Give students opportunity to apply language to their lives 
-Assess individual progress 

-Move around room to listen and provide assistance to individual pairs 

♦ Criteria for evaluating the effectiveness of the ESL Institute training: 
-Clear presentation of objectives 

-Effective use of audio and visual aids, 

-Teacher perceptions of understanding of concepts, preparedness to apply concepts 
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Additional indicators recommended by the evaluation team included: 



♦ Refinement of training design and content 

♦ A transfer process that can be used to prepare adult education providers for 
participation in the Institute 

♦ Reinforcement activities foi enhancing skill building and skill retention of Institute 
participants 

Participant Outcomes 

♦ Extent to which students participate in classroom activity 

♦ Extent to which students complete classroom activity 

Selection of Measures 

Three instruments were used to document participants' mastery and implementation of 
Institute training techniques: ESL Institute feedback forms, classroom observation form, and the 
Teaching Improvement Process. Interviews were also conducted with teachers whose classrooms 
were observed. Observers used the Institute's feedback forms for eight of 11 ESL techniques to 
document participants' abilities to perform the techniques. Classroom observation forms were 
also used to document instructor techniques, and the Teaching Improvement Process (TIP) was 
used to record instructors' implementation of CBAE management strategies. 
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California 

Evaluation of the First Year Implementation 
of Its Four Year ABE Plan 



Evaluation Overview 

This report outlines the results of an evaluation of the 1989-93 California Stale Plan for 
Adult Basic Education, conducted by the Evaluation and Training Institute, Los Angeles, 
California. The purposes of the study were to conduct a formative and summative evaluation of 
the implementation of the four major goals of the ABE Stale Plan: 



Improving access to users; 
Improving accountability; 

Improving program quality and responsiveness; and 
Improving planning and coordination. 



Seven major evaluation activities were conducted in the first year: 

♦ Formation of an Evaluation Advisory Committee; 

♦ Development of a matrix of adult education programs; 

♦ On-site peer review of 10 Section 321 grant recipients; 

♦ Written survey of all Section 321 and 326 grant recipients; 

♦ Telephone interviews with members of AE Steering Committee and 353 grant 
recipients; 

♦ Analyses of local assistance grant achievement data; and 

♦ Case study of state level activities. 
Measures of Program Quality 

Program quality in this instance reflects the implementation of the four goals noted 

above. 

Program Context 

♦ Increasing access through expansion of open entry/open exit competency-based 
education services to alternative modes of delivery; 

♦ Expansion of ABE programs to serve those whose primary language is English, 
with incentive funding available; and 

♦ More equitable access to funding throughout the state. 
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Program Process and Content 

• Expanded recruitment efforts; 

• Interagency involvement in outreach, referrals; 

• Improved coordination among literacy-related and basic skills service providers; 

• Expand key stakeholders in planning and oversight of programs; and 

• Expanded staff development activities. 
Program Outcomes 

• Use of CASAS system for pre- and post-testing. 

Funding Criteria 

The Department provided a 25 percent increase in rate structure per hour to encourage 
increasing services to ABE students who speak English as their primary language, thus comprising 
15 to 20 percent of the total numbers of people served. 
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Georgia Follow-up Study of GI£I) Graduates 



Kvalualion Overview 

This report profiles a 1989 study of GED graduates to determine what effect passing the 
GED Tests had on the participants' lives. The study focused on outcomes relating to 
employment, further education, personal satisfaction, and family well being. Other social and 
economic factors were examined, including respondents' expectations before and after passing the 
test, future plans, income changes, and changes in confidence level. One thousand students were 
randomly selected from a pool of over 10,000; 33 percent responded to the 19-item survey. Data 
were analyzed by grade completion, race or ethnicity, age, and gender. Of those surveyed, 99 had 
completed the 10th grade, 85 completed the 11th grade. 

Measures of Program Quality 

Program Context 

Demographic characteristics of participants; 
Motivation for taking tests; 
Employment status before taking test; and 
Expectations and future plans of program participants. 

Program Process and Content 

• Recruitment strategies; and 

• Type of preparation for test. 

Program Outcomes 

• Occupational status after receiving GED; 

• Income level after receiving GED; 

• Effects on respondents' self-confidence; and 

• Perception of family and social status after receiving GED. 

No discussion is offered as to methodology or rationale for selection of these criteria. 
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Illinois 

1989 Instructional Program Evaluation of ABE 



Evaluation Overview 

This report provides a descriptive analysis of Project Quality, the evaluation process used 
in Illinois. Eighteen programs received on-site evaluations conducted by peer adult educators 
under the leadership of an experienced adult education program evaluator. The evaluation 
process consisted of a review of written material provided by program administrators; self- 
assessment instruments which were provided to program administrators, teachers, and cooperating 
agencies; data from the Area Planning Councils; and the on-site visits. As part of the process, 
sites requesting funding for the following year are required to indicate how they arc responding to 
the evaluation recommendations for program improvement which were provided. The adequacy 
of the program's response is considered by the State in providing for continued funding of the 
program. 

Measures of Program Quality 

The bulk of the activities examined fell into the area of program process and content. 
Program Process and Content 

♦ Program Planning and Content 

Long-range program planning 

Interagency planning with other social service providers 
Increased student follow-up mechanisms 

♦ Qualifications of Personnel 

Full-time directors/instructional staff 
Support services for director 
Availability of staff development activities 
Needs assessment for staff development activities 

There was no discussion as to the selection of these criteria for review. 
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An Evaluation of Adult Literacy Efforts in Indiana 



Evaluation Overview 

Under the direction of the Indiana Adult Literacy Coalition, an evaluation was conducted 
to assess the status of adult literacy programming in Indiana, evaluate the impact of the Literacy 
Coalition, and develop a database of literacy providers. With funds from a Gannett Foundation 
grant, the Indiana Adult Literacy Coalition established an evaluation task force in 1987 to develop 
an evaluation design. A second task force guided further development and implementation of the 
design. Three surveys were conducted for the study: a provider survey of 116 ABE programs, 
volunteer organizations, private industry councils, and vocational technical colleges (representing a 
96 percent response rate); an anonymous survey sent to provider respondents two months later 
(58 providers responded, or 50 percent); and a local coalition survey conducted by telephone to 
all 27 literacy coalitions in the state. The study also involved visits to providers and local 
coalitions in 10 communities and observations of state coalition functions. 

Measures of Program Quality 

Program Context 

• Developing plan for adult literacy; 

• Census data and estimates of functional illiterates, high school non-completers; 

• Types of organizations providing literacy services (e.g., ABE programs, volunteei 
organizations, private industry councils, vocational technical colleges) and number 
of learners served by each organization; 

• Program longevity; 

• Provider support for and active involvement in local literacy coalitions; 

• Enrollment of learners by reading level, sex, age, ethnicity, employment status 
(current levels and changes); 

• Program funding sources (e.g., Federal, state, local/public, private); 

• Availability of materials and technical assistance to providers; 

• Promoting public awareness of literacy (e.g., hotline, newsletters, recognition 
programs, promote research, speakers' bureau); and 

• Percent of target population served. 
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Program Process and Content 
Program Promotion/Coordination 

• Literacy providers form partnerships with other organizations that share 
responsibilities of literacy programming (vary by type of organization, nature of 
partnership); 

• Fund-raising efforts; 

• Provider choice of priorities (e.g., publicity, instructional materials, tutor training, 
more paid staff); 

Staffing 

• Staffing configuration (e.g., number of part-time staff vs. full-time staff; paid staff 
vs. volunteers; staff positions such as administrators, teachers, counselors, 
paraprofessionals); 

• Volunteer referral/recruitment/activation; 

• Volunteer activities and characteristics (e.g., responsibilities, training and support, 
demographics); 

Program Operation and Services 

• Program response to learner demand (need for waiting list); 

• Program accessibility such as schedule (e.g., open-entry, open-exit schedule; hours 
of operation) and location (e.g., type of site, convenience to learner and 
instructor); 

• Instructional facilities; 

• Availability of support services such as transportation, child care, and counseling 
services; 

• Collaboration with private businesses to provide on-site workplace literacy 
instruction; 

Instruction 

• Time spent in literacy instruction (contact hours, homework); 

• Instructional approach (e.g., one-to-one tutoring, small group instruction, learning 
center, classroom instruction); 
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• Instructional materials and methods (e.g., workbooks, computers, basal readers: 
phonics, language experience); and 

• Testing (availability, function, type of testing). 
Participant/Program Outcomes 

• Learner retention; 

• Literacy instruction more accessible to learners; 

• Improved quality of instruction; 

• More learners enroll "who otherwise would not have"; 

• More volunteers participate "who otherwise would not have"; 

• In-kind resources are committed to literacy efforts; and 

• Private funds are raised for literacy. 
Selection of Measures 

The focus of the surveys and the evaluation in general was on the level of involvement of 
literacy providers in literacy coalitions to promote literacy statewide. The lengthy list of measures 
above provide a descriptive look at literacy programs in the state rather than a prescriptive 
approach for determining quality indicators. However, a few of these items were highlighted as 
important indicators of the positive impact of literacy efforts: increases in enrollment, use of one- 
to-one volunteer tutoring, program participation in literacy coalitions, and providers' awareness 
and use of promotional services (e.g., publications, referrals, conferences). 

The process for the selection of these indicators does not appear to have been a 
structured one. Evaluation designers utilized information from Census data, the evaluation task 
force, and interviews with literacy coalition members and provider staff. 
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Iowa 

Adult Basic Education Assessment 

Evaluation Overview 

The state of Iowa undertook a statewide assessment to: 

(1) Identify all the assessment procedures and practices being utilized by the ABE 
program; 

(2) Identify the target populations served; 

(3) Identify major assessment outcomes cross-referenced by target populations; 

(4) Develop a taxonomy of major assessment categories; and 

(5) Recommend additional assessment procedures or practices to be utilized on a 
statewide basis. 

Measures of Program Quality 

The concept of assessment focused on student progress, achievement or learning 
outcomes, but success is based on what the learner's goals are -- what he or she wants to know. 

The following were the major assessment categories identified and supported by the 
state's ABE program: 

(1) Testing to include norm-referenced, teacher-made tests and criterion-referenced 
tests; 

(2) Student goal-setting performance measures which include anecdotal records, 
follow-up procedures, and pre/post assessment procedures; and 

(3) Other performance measures such as personal observations, role playing, oral 
presentations, writing assessment, video feedback, and life simulation scenarios. 

Selection of Measures 

It was determined that, for pre- and post-assessment on standardized tests, either the 
TABE or ABLE be utilized. 
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Externa! Evaluation of the Kansas Adult Education Program 
for Fiscal Year 1990 

Evaluation Overview 

This evaluation of the Kansas adult education program examined existing evaluation and 
reporting systems used by the state's local programs, summarizing and assessing the kinds of data 
they collect, and developing a comprehensive system for evaluating adult education programs in 
Kansas. The primary objective of the evaluation was to make recommendations for evaluation 
procedures to be used in the succeeding three years. Evaluators contacted local program 
directors through a written survey, a presentation at a statewide meeting, letters requesting 
program information and providing status updates of the evaluation, and follow-up phone calls. 
Through these channels, evaluators obtained summary descriptive statistics from 12 local programs 
(e.g., program description, test scores), as well as survey responses from 36 of 57 state directors 
(63 percent). 

Measures of Program Quality 
Program Context 

• Presence of a state advisory council for adult education; 

• Information on local area, e.g., industry, economy, employment; 

• Categorization data on program participants (e.g., ESL, institutionalized, homeless, 
unemployed); 

Program Process and Content 

• Personal contact with schools and social agencies; 

• Common evaluation and reporting system across local programs; 

• Needs assessment, statement of priorities established under the state plan; 

• Use of placement or diagnostic tests (e.g., WRAT, TABE, GED Practice Tests) at 
entry, periodically throughout the year, and at exit; 

• Statewide, standardized assessment instrument; 

• Proper test administration and scoring; 

• Program operates in a convenient location for students; 

• Curriculum planning documentation; 

• Program implementation data (e.g., number of open-entry centers, number of 
classes taught, number of students per class, number of hours per class); 
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Computer capacity (for administration or instruction) at ABE/GED centers; 
Provide counseling to participants; and 
Follow-up when a client stops coming to the program. 
articipant Outcomes 



Post-test achievement; 
Work-related outcomes; 

Increased student proficiency in reading, writing, speaking, listening, computation, 
problem solving; 

Reduced dependency on welfare; 

Entering training programs such as vocational education and JTPA; 
Student reporting on objectives met; 

Student feedback to teachers and counselors reflecting students' perception of how 
effective program has been to them; 

Information from program leavers and graduates relating to adequacy of 
instruction and their ability to pursue further education and training; 

Feedback from individuals and organizations (including employers) who have an 
interest in continuation of adult education programs; and 

Award scholarship or certificate to GED high scorer or graduate. 
Selection of Measures 

The measures of program quality, which appeared in the survey of directors and an 
information form that each director completed for his or her program, are drawn largely from 
items listed in the annual performance report The state director for ABE and GED compiled a 
list of questions, as did some local program directors. 
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Louisiana 

On-Site Evaluation of 66 Local Adult Education Programs 



Evaluation Overview 

Sixty-six local adult education programs were evaluated during 1989-90 by the supervisory 
staff of the Bureau of Adult and Community Education of the Louisiana Department of 
Education. Evaluation (data collection) questionnaires were designed to elicit information on 
characteristics of the program, with specific focus on the issues of recruitment, cooperating 
agencies, and staffing. 

Measures of Program Quality 

Program Context 

• Types of programs; and 

• Location and scheduling. 
Program Process and Content 

• Recruitment strategies; 

• Outreach efforts; 

• Cooperative agreements with other public agencies; 

• Involvement of the private sector in recruitment strategies; 

• Support services including child care and transportation; 

• Staff characteristics: 

Full- or part-time 
Certified in adult education 
Professional qualifications 
Staff development activities 
Evaluation of teachers 

• Instructional methodologies: 

Individualized instruction, and 
Diagnostic profiles of students. 

There is no discussion as to why these measures were examined. 
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Michigan Adult Literacy Initiative Project 



Evaluation Overview 

At the request of the Adult Extended Learning Services of the Michigan Department of 
Education, Formative Evaluation Research Associates (FERA) conducted an evaluation of the 
Michigan Adult Literacy Initiative (MALI) Project. MALI, organized into 14 regions, coordinates 
statewide activities aimed at reducing adult functional illiteracy. The primary purpose of the 
evaluation was to determine the "overall effectiveness of the project in increasing the literacy 
rates of adults in Michigan." The evaluation also addressed questions related to building 
community awareness, training tutors, developing successful programs, and organizing MALI. 

Measures of Program Quality 

Program Context 

• Number of students and tutors involved in literacy programs; and 

• Reputation of program among local agencies and organizations. 
Program Process and Content 

• Pattern of volunteer involvement in the program; 

• Effect of the program on student progress; 

• Ratio of staff to tutor/student matches; and 

• Effect of student ability on tutor performance and retention. 
Participant Outcomes 

• Student perceptions of success from participation in a program; 

• Number of students attaining their goals; 

• Impact of tutorial on attitudes of families and employers; 

• Impact of services on local community; 

• Extent to which "tutors make a difference" at ABE sites; and 

• Impact of tutorial on student self-esteem. 
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Selection of Measures 



Measures were selected based on data collected from the following sources: ( 1 ) a review 
of documented data from each region; (2) interviews with the statewide coordinator of SDC and 
regional facilitators; (3) interviews with groups of literacy tutors, ABE instructors, employers, and 
students in three local sites; (4) creation of exit and follow-up interview forms to be used by 
volunteers to collect student information; (5) interviews with administra' ors of three local literacy 
programs; an i (6) creation of a computer-based management information system to monitor 
tutors and kiudents across he stale. Information for the evaluation design was also based on 
input from regional facilitators and the state director of adult education and his staff. 
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Staff Development Collaborative Project 
Michigan 

Evaluation Overview 

At the request of the Adult Extended Learning Services of the Michigan Department of 
Education, Formative Evaluation Research Associates (FERA) conducted an evaluation of the 
state s Staff Development Collaborative (SDC). The primary goal was to determine whether 
participants achieved objects of the staff development activity and whether participants are using 
what they learned three to six months after training. Evaluation methodology included interviews 
with the statewide coordinator and all regional facilitators; analysis of the SDC workshop 
evaluation database in the State Department of Education; a mail survey of participating and 
nonparticipating adult educators; and a mail survey of a sample of participating and 
nonparticipating school districts. 

Measures of Program Quality 

Program Context 

♦ Administrator perceptions of the value of staff development and the SDC 
Program Process and Content 

♦ External linkages to school districts and teachers (e.g., financial and philosophical 
support from school system, adult educator networking); and 

♦ Professional development content 

Meeting professional development needs of adult educators 
Effectiveness of workshops in meeting needs 
Participant exposure to current trends, models, materials 
Realistic goals and objectives of workshops 

Participant perceptions that the workshop was helpful three or six months 
later. 

Selection of Measures 

FERA met with SDC regional facilitators to ask about their activities, goals, and 
evaluation questions. Responses from the facilitators were used to design the evaluation. 
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Evaluation of Adult Refugee ESL Programs in Minnesota 



Evaluation Overview 

The Minnesota Department of Education commissioned an evaluation of the state's 18 
adult refugee ESL programs funded in 1985-86 through the Minnesota Refugee Program Office 
of the Department of Human Services. The purpose of the evaluation was to define and measure 
the effectiveness of the 18 refugee programs, identify successful classroom practices, and make 
recommendations for future decision making and planning. Sources of information used to 
conduct the evaluation were the Refugee Program Office's data system and information from 
participants and service providers. Data collection and evaluation design activities included the 
following: (1) six design meetings with the Adult Refugee Coordinator at the Department of 
Education; (2) a review of project files and relevant literature; (3) a preliminary meeting with 
project coordinators; three structured meetings with representatives of the State Refugee Program 
Office and the Department of Education; (4) site visits to 16 of the 18 programs, including 
interviews with more than 50 teachers; (5) five student focus groups, with input from more than 
50 students; (7) a focus group with teachers; (8) and a review of demographic and program data 
from the Refugee Program Office. 

Measures of Program Quality 

Evaluators identified a number of contextual and method factors that appeared to be most 
important to program effectiveness. The degree of emphasis varies somewhat depending on 
student performance levels. 

Program Process and Content 

• Attract new refugees who have never attended ESL classes in the agency's 
community (e.g., relax enrollment regulations, use culturally comfortable setting, 
arrange for transportation, actively coordinate with sponsoring and resettlement 
agencies; 

• Support retention of students through support services (e.g., transportation, child 
care, ability to address crisis resettlement problems, translation services, bilingual 
staff, access to other bilingual resources); 

• Hire and support excellent teachers; provide opportunities for training, workshops, 
idea and materials exchanges; 

• Recruit program coordinators with leadership skills, and strengthen the 
coordinator's role; 

• Promote coordination with other agencies; 

• Structure programs that can adjust to students* individual, educational needs (e,g., 
open enrollment, classroom aides and volunteers, in-home tutors); 
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♦ Maximize instructional time and pace (e.g., better retention with daily classes, 
three to tour hours; limit classes to 20 students, preferably 10); 

♦ Organize classes by student performance levels; 

♦ Address student expectations in classroom design and process (e.g., discuss 
personal goals, use written materials for homework, use books and organize 
materials that give students a sense of ownership and progress); 

♦ Actively involve students in classroom activities; reduce teacher-directed activities 
as a class progresses; increase individualized attention as needed; 

♦ Teach students to learn (e.g., develop study skills, problem solving, encourage 
questioning); 

♦ Use relevant issues as teaching tools (e.g-, discuss homeland, offer field trips, 
discuss job opportunities); 

♦ Encourage and enhance learning outside of the classroom (e.g., assign homework, 
practice); 

♦ Involve students in their educational goals and the design of classes; and 

♦ Balance use of appropriate materials (e.g., use brief conversation in dictation, 
grammar, memorization, oral recitation). 

Selection of Measures 

Outcome data on student progress and language level correlation was unavailable. 
Evaluators found that teachers and students had difficulty articulating what elements made their 
programs successful. Evaluators also indicated that system design problems were more significant 
than method selection in determining the effectiveness of ESL instruction. However, in the 
absence of outcome data, evaluators relied on classroom observations and interviews with teachers 
and students. 
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Impact Evaluation of Adult Basic Education Program Outcomes 

Montana 

Evaluation Overview 

The study of ABE participants in Montana involved a research and evaluation component, 
including a review of research studies that have examined student outcomes and voluntary 
interviews with ABE students. The primary purpose of the study was to describe the qualitative 
(e.g., improved self-concept) and quantitative (e.g., financial gain) effects ABE programs have on 
participants* lives. Based on a random sample of nine of the state s 19 ABE programs, 516 ABE 
participants were interviewed (328 former students and 188 current students). Data were 
collected at program sites by trained teams of interviewers in half-hour interviews with each 
student. Data from these interviews were later used to assess and evaluate program impact on 
students' lives. 

Study findings indicated that the vast majority of both current and former students who 
were interviewed felt that participation in the ABE program had a "major, positive impact" on the 
improvement of their basic skills and selected applied knowledge areas, and increased the 
likelihood of their participation in community activities and their children's education. Whereas 
interview responses suggest that there may be an association between participation in ABE and 
positive changes in their students' lives, interview data are not sufficient to conclude that ABE 
programs caused these changes. 

Program Context 

• Characteristics of former vs. current students (e.g., sex, age, ethnicity, residence, 
number of children, educational attainment); and 

• Reason for attending ABE programs. 
Program Process and Content 

• Student perceptions of best features of ABE programs (e.g., self-paced instruction, 
freedom for personal growth, learning basic skills, program structure, teachers); 

• Student perceptions of worst features of ABE programs (e.g., individual personal 
problems, single program component, individual classroom experiences); and 

• Student suggestions for program improvements (e.g., improve teaching, individual 
concerns, improve classes). 

Participant Outcomes 

Changes in Skills, Knowledge Areas, Personal Life 

• Achievement of personal goals; 

• Improvement in basic skill areas (e.g., math, reading, writing, and communication); 
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• Improvemeni in basic knowledge areas (e.g., how to find and keep a job, 
knowledge of your rights and responsibilities as a citizen, knowledge of health 
care); 

• Involvement in community activities (e.g„ use of library, vole in elections, outside 
reading, participate in community organizations); 

• Involvement in formal education of their children (e.g., attend parent meetings at 
school, talk with teachers or principals about their children); 

• Overall change in basic skills, knowledge, and community involvement; related to 
basic skills improvement, self-awareness and personal growth, and relationships 
with teachers and staff; 

• Enrollment or intention to enroll in other educational programs while attending, or 
after terminating from, ABE programs; 

• Status (enrollment, completion, termination) in other educational programs such as 
college, vocational/technical school, GED preparation; 

• Impact of ABE program attendance on participation in other educational 
programs; 

• Changes in personal life (e.g., sense of satisfaction with life, getting along with 
other people, level of self-confidence, way you feel about yourself); and 

• Changes in relationship with children and children's social and educational 
development. 

Financial Gains 

• Receipt of/dependence on government financial assistance; 

• Employment status and employment income; 

• Family income, condition of family finances; 

• Program cost per student hour of instruction; 

• Hours of ABE program instruction; and 

• Ratio of net income changes for individual students and families to ABE program 
costs. 
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Selection of Measures 

Outcome measures were based on findings from the research literature and input from a 
project advisory committee, which consisted of program administrators from the sample programs 
and administrators from several other programs. These measures were analyzed based on student 
answers to a structured interview instrument that included questions in the following areas: 
reasons for attendance, educational impact, skills acquired, personal changes, program evaluation, 
economic benefits, and demographic information. 
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Learning in Literacy Volunteers of New York City 

Evaluation Overview 

This mostly qualitative evaluation study, conducted in 1990, assesses the impact of student 
participation in the Literacy Volunteers of New York City (LVNYC) program. Program data 
were collected primarily through open-ended focus group interviews, individual interviews, and 
observation. Additional data that were collected included program documents, students' 
demographic information, standardized test scores, and writing samples. Program literature 
included the tutor training manual, student advocate guidelines, past internal and external 
evaluations, data forms, and minutes of the Student Committee meetings. Evaluation staff 
interviewed 102 students (65 in focus groups and 37 by phone). Ten students were selected from 
the focus groups to serve as case studies. Twenty tutors were interviewed, 16 staff members were 
interviewed, and 20 observations were conducted at the LVNYC during tutor training, pre- 
enrollment sessions, instruction, and at an annual program-wide event. Students were asked to 
describe how they came to the program, their goals, and changes they had seen in themselves. 
The evaluation examined student changes in three general areas: literacy skills; self-concept, 
attitudes and beliefs related to literacy development; and involvement in literacy tasks outside the 
program. 

Measures of Program Quality 
Program Context 

• Prior schooling experiences of students; and 

• Student goals, motivations for enrolling (commitment to children, further 
education, employment-related reasons). 

Program Process and Content 

• Student-centered learning that enhances students' literacy skills and impacts their 
self-esteem 

Help students move from a text-based to a knowledge-based approach to 
reading 

Extensive use of writing for instruction and personal reflection 
Use of group instruction 

Students identify their goals, choose reading materials, write on topics 
reflecting their interests and goals 
Commitment to student leadership development 
Programs celebrate students' reading and writing 

In addition to what evaluators found in the program, several elements were recommended 
for program effectiveness: 

• Systems of tutor support to decrease attrition and enhance effectiveness of 
instruction; 
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• Program policy addressing students* rights in relation to tutor changes imposed by 
Center directors; student handbook explaining students* rights ; 

• System for regular in-service staff development; 

• Aggressive steps to establish cultural diversity of staff and volunteers; include 
cross-cultural communication as part of tutor training; 

• Counseling in a variety of formats; 

• Program support for students' use of skills outside the program (e.g., encourage 
classmates to accompany one another on outings such as grocery shopping, job 
hunting, bank transactions); 

• Student involvement in decisions about instructional groups and tutor assignments; 
and 

• Ongoing assessment, evaluation, and program development; involve stakeholder 
groups in process; develop center, personal, and central records (relying on 
alternative assessment model using portfolios). 

Participant Outcomes 

• Activities outside program 

Personal reading and writing at home 

Read to oneself in public (e.g., posters, signs, labels) 

Represent program to media 

Fill in job application 

Open a new bank account 

Formal correspondence 

Paying bills 

Shop for groceries 

Activities inside program 

• Personal reading and writing in tutoring group; 

• Read feedback from fellow students at a celebration, read to visitors in front of 
tutoring group; 

• Results on standardized tests; 

• Assist in junior training and student orientation; and 

• Practice filling in applications in tutoring group. 
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Selection of Measures 



Measures were based on the LVNYC program philosophy, research literature on qualities 
of learner-centered programs, and many student-generated criteria. Criteria that were selected 
were based on a qualitative approach to examining indicators of student change and improvement. 
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Starting Oven Characteristics of Adult Literacy Learners 
Literacy Assistance Center, New York City 

Evaluation Overview 

This case study followed three studies of adult literacy learners in New York City that 
were commissioned by the Literacy Assistance Center in 1986 for the New York City Adult 
Literacy Initiative. The purpose of the study was to provide more detailed, descriptive 
information about the educational and life histories of native-born and native-English speaking 
adults who read below a 5th grade reading equivalency. The first phase of the study was a ase 
study consisting of one-hour structured interviews of a sample of adult education students in New 
York City literacy programs. Fifty-eight individuals were interviewed; data from 32 of these 
students were included in the analysis. Study staff also utilized an analysis of a city-wide database 
of student characteristics. Phase II of the study was to build on these interviews through an 
ethnographic follow-up. 

Six research questions guided the study: 

1. How do adults find out about literacy programs? 

2. How easy or difficult is it to gain entry? 

3. What makes adults decide to attend? 

4. What factors influence attendance/dropout rates? 

5. What is the background of participants? 

6. Significant events in participants' lives, how do they relate to educational situations? 
Measures of Program Quality 

Within the six broad areas, interview questions covered the following topics: 

• Life situation (current residence, reading and writing experiences outside of 
program, community involvement); 

• Past life and educational experiences (previous schools, when left school, 
awareness of literacy abilities in past, parents 1 and siblings' educational 
experiences/literacy, reading materials in the home, language spoken in the home); 

• Prorram participation (reasons for attendance, how they heard about the program, 
difficulties in getting to class, help with schoolwork, participation in choosing what 
to learn, indicators of making progress/meeting goals, changes in life since 
enrolling in adult education program); 

• Job history (current employment status, previous jobs held, opportunity to get a 
better job, ability to handle reading or writing tasks on the job, future job plans). 
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Selection of Measures 



Interview questions were based on a review of the research literature. The six basic areas 
covered in the interview questions were chosen to elicit specific information about the 
characteristics of adult literacy students, centered around three areas: student motivation for 
enrolling in adult programs, personal and professional goals, and reasons for continuing or leaving 
programs. The study provides implications for ways in which programs can effectively respond to 
the needs and perceptions of adult learners and issues related to student recruitment and 
retention. 
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North Carolina Program Evaluation and Implementation Report 
Evaluation Overview 

The North Carolina Department of Community Colleges awarded a grant to the North 
Carolina Commission of Indian Affairs in 1983 to increase Indian participation in ABE programs 
in four counties. An evaluation was conducted to determine the effectiveness of local programs 
in increasing Indian participation. A total of 135 Indian adults were recruited in these programs, 
120 more than anticipated for the year. The retention rate for the project was more than 50 
percent. 

Measures of Program Quality 
Program Context 

• Participant background (e.g., family status, sex, receive public assistance, family 
income, last school and grade attended) 

Program Process and Content 

• Recruitment program that targets a specific group; 

• Utilize Indian instructors and aides; 

• Locate classes away from community colleges; based on belief that Indian ABE 
students would be uncomfortable in a school setting, gather more support from 
classmates they know; 

• Recruiter/counselor met with each participant to identify personal, educational, 
and career goals, and planned field trips to make students aware of employment 
opportunities available in the community; and 

• Strong support system and encouragement between students, teachers, and 
recruiter/counselors. 

Participant Outcomes 

• Student goals and objectives; 

• Retention rate; 

• Obtain high school equivalency; 

• Increase in participant job readiness and marketability; 

• Participant recruitment of family or friends to attend ABE classes; 



78 



Enrollment in a community college vocational program; and 



• Obtain job. 
Selection of Measures 



Indicators of quality appeared on the application/intake form filled out by participants and 
a worksheet on which they listed goals and objectives. It appears that many of the indicators are 
pulled from typical intake information or outcomes associated with student goals. The report 
docs not indicate how the Commission on Indian Affairs determined what recruitment and 
retention efforts worked best for Indian students. 
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North Carolina ABE Instructional Program Evaluation, 1985 
Evaluation Overview 

A team of researchers led by Hanna Fingeret conducted an extensive evaluation of North 
Carolina's ABE programs using a multi-site case study model (adapted from Stake's responsive 
evaluation). The study included a literature review of outcome and impact studies, a feasibility 
study, a survey, on-site and telephone interviews, and collection of local and state program data. 
Initially, a statewide survey was conducted to determine whether or not there was support for a 
large scale evaluation; 296 of 511 "stakeholders" identified by local program directors responded 
to the survey. Evaluation staff asked the survey respondents to respond to evaluation designs. 
The original design was to collect existing numerical data describing demographic characteristics of 
ABE students, participants' progress and length of participation in the program, and recruitment 
and follow-up data. In fall 1983, a feasibility study was conducted to test design and develop 
interview/observation protocols. For the case study, six teams collected information from taped 
interviews with 166 participants and staff from six programs chosen to represent rural and urban 
areas and demographic and racial mixes of students. Interviewers collected program documents 
and records in addition to observation and interview data. A large part of the report is a 
narrative of anecdotes from interviews with students and instructors. 

Evaluation staff tried to ensure the validity and reliability of data through several 
measures: use of multiple researchers working independently; use of local observers to sit on site 
teams; multiple interviews with students from the same classroom; combining interviewing and 
observation of instructors; and independent researcher data analysis of individual sites compared 
with team member's site analysis and principal investigator's cross-site analysis. 

Measures of Program Quality 

Program Context 

• Structure of state ABE system: funding and record keeping emphasis, isolation vs. 
autonomy, identification of problems, leadership; 

• Student characteristics (prior schooling, employment, life experiences, goals, life 
cycle stage) and their relationship to goals; 

• Program administrators' perspectives on purpose of ABE, administrative 
philosophy, program models, linkages, sites and facilities, recruiting, program 
expansion; 

• Instructors' background (how they got involved in ABE, education, professional 
development and learning on the job, perspectives on student motivations and 
background, perspectives on purposes of ABE); and 

• Reasons for enrolling in programs (educational vs. employment reasons). 
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Program Process and Content 



♦ Instructors* perspectives on their responsibilities (e.g., creating personal 
relationships with students, meeting individual emotional and psychological needs, 
"keeping" students, helping students meet individual goals); 

♦ Resource constraints (administrators' perspectives on hiring instructors, instructors' 
status, procuring instructional resources, availability of instructor training); 

♦ How success is defined by students, instructors, and administrators; 

♦ Collection of quantitative program data (e.g., follow-up data on participants); 

♦ Instructional approaches for reading that focus on meaning and context; and 

♦ Use of interactive instructional approaches (rather than bottom-up, top-down, 
subskill). 

Participant Outcomes 

♦ Boost in self-concept, self-confidence; 

♦ Definition of success or completion; 

♦ Students tended to stay longer when the teacher considered how and what they 
wanted to learn; 

♦ Reading and writing skills; and 

♦ Increase in employment. 
Selection of Measures 

A number of issues emerged from the case study analysis: the need for a clear definition 
of success or completion of ABE; the lack of ABE program identity (lack of distinction between 
ABE and GED students); lack of teacher training; and lack of useful quantitative data, especially 
lack of follow-up data. Researchers noted that the research and evaluations tend to focus on 
individual outcomes instead of trying to understand the interaction between the program, student, 
and environment. An impact and outcome focus fails to get at how adults participate, remain, or 
leave programs. Qualitative or naturalistic research methods (e.g., interviews and observations) 
are more appropriate for responsive evaluations. 
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North Carolina 
An Evaluation of ABE Programs 



Evaluation Overview 

This report profiles the procedures and findings of an evaluation of the ABE program of 
the North Carolina Community College system conducted by the Center for Urban Affairs and 
Community Services of North Carolina State University. The focus of the study was to provide 
basic descriptive information on the following features of ABE programs: program funding, 
program administration, educational and support services, instructional methods and materials, 
suggestions for the improvement of service delivery, and overall assessments of the ABE program. 
ABE directors, instructors, and students were surveyed in questionnaires designed in conjunction 
with the North Carolina Department of Community Colleges. Overall recommendations from 
administrators and teachers included a need for better methodology and resources devoted to 
recruitment and program publicity; retention activities; increased staff development activities for 
instructors and higher pay levels; reduced class sizes; and more flexibility in funding formulas. 
Student recommendations also supported reduced class size, more individualized instruction, 
increased program publicity, and increased support services such as child care and transportation. 

Measures of Program Quality 

Three surveys that were utilized reflected different aspects of the program. 
Program Context 

• Funding and program expenditures; 

• Types of educational services offered; 

• Location and scheduling of services provided; 

• Support services offered; and 

• Student demographics. 
Program Process and Content 

• Types of instruction provided; 

• Program staffing; 

• Staff development activities including in-service and pre-service; 

• Teaching hours per year; 

• Types of instructional materials; and 
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♦ Effectiveness of various program components. 
Participant Outcomes 

♦ Reasons for enrollment; 

♦ Specific ABE services needed; 

♦ Assessment of program effectiveness; and 

♦ Need for support services (transportation and child care). 
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Adult Education in Texas Program Evaluation, FY 1986-88 



Evaluation Overview 

This statewide evaluation of adult education programs in Texas examines "the extent to 
which provisions of the Texas State Plan for Adult Education Plan for Fiscal Years 1986-88 have 
been met" The study involved three activities: (1) an analysis of extarA individual program data 
from documents such as program applications, annual performance reports, rules, regulations, and 
the TEA monitoring instrument; (2) mail surveys and interviews with students, teachers, program 
administrators, and representatives of linkage organizations; and (3) a summary report. About 
half of the state's 63 program directors responded, as did 595 teachers from 54 cooperatives. A 
stratified sample of adult education students was developed based on enrollment size of 
cooperatives, geographic location, and level of class (ABE, ESL, GED); 1,461 students responded. 

* Students were surveyed to elicit their perceptions of teacher availability for conferences, 
instructional materials being used, skills being acquired, and objectives being met. Questions on 
the teacher survey examined types of instructional materials and assessments used, extent of the 
use of volunteers in classrooms, type of technical assistance received, awareness and use of 
information from special projects, and perceived program strengths and weaknesses. Leaders of 
community linkage organizations were interviewed to determine the extent of their cooperation 
with adult education programs. Program directors were interviewed to determine their use of 
special project information, extent and quality of technical assistance and stt " development 
received from TEA, and their perceptions of strengths and weaknesses of local programs. 

Measures of Program Quality 

Program Context 

• State staff provide leadership in local program management and planning (e.g., 
train local program administrators in the development of long-range and annual 
program plans, help them develop appropriate program goals and policy; and 

• Statewide network of information and referral for organizations interested in 
literacy programs. 

Program Process and Content 

• Utilize alternative learning strateg ies that emphasize individualizing instruction for 
"the least educated most in need"; 

• Use standardized curriculum with specific guidelines, scope, and sequence, 
particularly competency based; 

• Staff development 

Priorities based on program needs as identified by needs assessment from 
staff, students, administrators 

Coordinate training with higher education institutions 
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Disseminate teaching ideas to teachers through channel such as 
newsletters, teacher-made materials, catalog resources 



Sufficient funding to purchase instructional materials, pay teachers, provide 
adequate number of classes to meet demand, provide adequate facilities; 

Instruction 



Incorporate computer-assisted instruction 

Emphasize survival language in ESL, while also meeting needs of higher 
level students 

Provide pre-vocational skills/job training 

Incorporate daily life experiences into reading and math objectives 

Teachers provide individual assistance, vary instructional methods to meet 

the needs of individuals and the number of individuals 

Use of group activities and allowance for students to study at their own 

pace in chosen subject areas 

Friendly, relaxed classroom atmosphere 

Use latest, most effective strategies in whole language acquisition 

♦ Instruction al materials (e.g., should be current and varied; availability of 
individualized study materials, multilevel materials; use of audio-visual equipment); 

♦ Offer counseling services, including personal counseling and assistance/guidance for 
students to move from one educational level to another; 

♦ Develop or expand community linkages with public and private organizations (e.g., 
industry, public schools, community resources) to maximize outreach through 
referrals, providing space for classes, and providing volunteers and resources; 

♦ Use volunteers to provide individualized (especially one-to-one) instruction; 
encourage volunteer efforts; provide technical assistance to local programs to help 
them recruit, screen, train, place, and retain volunteers; 

♦ Assessment should be ongoing and should guide instruction; 

♦ Programming should be geared to special adult populations (e.g., institutionalized, 
physically disabled, older adults, mentally retarded); 

♦ Monitor and evaluate local adult education programs to improve programs; and 

♦ Program management (e.g., utilize automated management information system; 
train new directors to maintain program quality). 

Participant Outcomes 

♦ Pride in achievement; 
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Awareness of, interacting successfully in, the community; 



Helping children with school work and activities; 



Motivation, desire to learn; 



Improvement in academic skills (e.g., reading, writing, communication); 



Goals met; 



Seek further schooling; 
Obtain job/better job; and 
Attain citizenship. 



Selection of Measures 



Evaluation questions were developed for each of nine broad goals identified in the state 
plan. Specific questions in each goal area were developed by examining extant data in annual 
performance reports and monitoring guides (which provided a framework for expectations of local 
programs), and analysis of program applications. 
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Comprehensive Evaluation of the Statewide Texas Adult Basic 
Education Program 

Evaluation Overview 

This summative and formative evaluation was conducted to assess the effectiveness and 
impact of the ABE program in Texas and to plan for and suggest improvements. Extensive 
surveys, open-ended interviews, and on-site observations were conducted to gather data from 
teachers, students, administrators, state agency personnel, advisor)' board members, and linkage 
agencies. A random sample of 350 of the state's 1,800 teachers was surveyed; each teacher was 
also given six questionnaires to distribute to current and former adult education students; and 
questionnaires were mailed to a random sample of more than 1,100 community agencies expected 
to link services with ABE programs. A "naturalistic" evaluation design was developed to be 
sensitive to the students' views of the program and their reasons for participation. 

Measures of Program Quality 

Program Context 

• Program identification of the hard-to-reach and their major needs and interests; 
and 

• Student characteristics (age, race, sex, program enrollment, educational 
background, family background). 

Program Process and Content 

• Program understanding of "stopping out" by students; mechanisms that make it 
easy for students to return to a program; 

• Staff input into program planning; 

• Linkages/programmatic ties with other community agencies, especially those serving 
the same clientele; 

• Quality of teaching (e.g., teacher competence in subject matter, turnover rate of 
teachers, teacher personality traits that enable them to relate to students, 
dedication and commitment of teaching staff); 

• Teacher benefits and recognition (e.g., salary, health insurance, career ladders, 
recognition of tenure); 

• Pre-service and ongoing staff development activities that stress methods useful in 
working with a mature learner and increased competencies in relating to the adult; 

• Recognition of and training for teacher aides; 

• Variety of program offerings to meet local needs/priorities; and 
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Strong supporting relationship wilh a strong supportive teacher; see adult 
education as a "human enterprise 11 . 



Participant Outcomes 



Increased self-confidence 



Selection of Measures 

Data collected for the evaluation varied depending on the group interviewed or surveyed. 
A two-part questionnaire for administrators asked about demographic information on local 
programs, planning, management, recruitment, instructional staff, linkages, and the instructional 
program. The teacher questionnaire consisted of 36 items in the areas of personal background, 
curriculum formation, classroom operations, students, materials, evaluation, facilities, and staff 
development; the teacher interview protocol contained similar topics, as well as instructional staff, 
students, planning, management, and linkages. The student questionnaires and interviews 
contained items related to reasons for participation, personal gains from the program, self- 
description, and assessments of the instructional staff, facilities, and program. The community 
agency questionnaire focused on class offerings, perceptions of the need for a local ABE program, 
extent of past cooperation, evaluation of the success of the local program, and suggestions for 
improved ABE services to the community in the future. 
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Texas 

FoUcm-up and Impact Study of ABE/ASE Students 



Evaluation Overview 



This report profiles a study completed in 1985 to assess the impact of adult education 
programs on the lives of completers of a GED or Adult Secondary Diploma, utilizing a sample of 
1,000 students who had completed their studies three years earlier (1981). The study examined 
the impact of these programs on students' lives in four major areas: economic/employment- 
rclatcd issues; responsibility and community involvement; further education; and personal growth. 
Surveys were returned by 955 students in all areas of the state. A 35-question survey was utilized 
as well as some case study analysis. This study reflected very favorably on the Texas Adult 
Education Program. Among some of the conclusions reached: (1) completion of an adult high 
school or GED Diploma program seemed to enable individuals to find employment and increase 
earnings; (2) for individuals desiring employment, the adult education experience enabled those 
with jobs to retain them and assisted most of those who were unemployed to find employment; 
(3) adult education seems to foster good citizenship, community involvement, and increased 
parental involvement in children's education; (4) most participants felt they had accomplished 
their goals in coming to the program. 

Measures of Program Quality 

This study focused on program outcomes. 
Program Outcomes 

♦ Employment Issues 

Pre/post employment/unemployment 

Change in occupational categories (utilized DOT categories) 
Salary changes 

Activities related to employment: use of want ads, increased responsibility, 
promotions 

♦ Individual Responsibility/Community Involvement 

Voting 

Driver's license 
Citizenship 

• Educational Issues 

Involvement in children's education 
Aspirations for children 

Education plans and achievement of adults including training programs 

• Personal Growth 

Increased confidence 
Involvement in current events 
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Program Evaluation 

Program's ability to satisfy needs of students 



Selection of Measures 

The topics and surveys were developed by an Advisory Committee consisting of a cross 
section of 27 adult educators from the state and six program staff. This committee included adult 
education teachers, co-op directors, Regional Service Center personnel, Community College staff, 
Texas Educational Agency staff, university consultants, and professional research and statistical 
specialists. 
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Evaluation of the Virgin Islands Adult Education Program 
Evaluation Overview 

The work of this evaluation was organized around four areas: program organization and 
structure, program operation and costs, program participants, and program outcomes. The study 
involved the following activities: (1) interviews with program administrators and program staff to 
learn more about the program's objectives, organization, and services; (2) on-site observation of 
instruction at a sample of program sites; (3) collection of data on program participants, the extent 
of program participation, and program outcomes; (4) collection of data on program costs; (5) an 
analysis of program services and program costs; and (6) preparation of a final report that includes 
recommendations to improve the organization and financing of the adult education program. 

A sample of eight schools was chosen for the analysis of the organization and structure of 
adult education programs, based on availability of all adult education components (ABE, ESL, 
GED, continuing education), diversity of program size, and representation from all three islands. 
A survey was sent to program coordinators at each of the eight schools. Information from the 
surveys was aggregated and analyzed to describe similarities and differences in programs' structure 
and organization. Data on participant characteristics were extracted from student records at the 
eight schools and entered into an individual student record database for analysis. 

Measures of Program Quality 

Program Context 

• Characteristics of program participants (age, gender, race, employment status, and 
citizenship). 

Program Process and Content 

• Formal program planning process; 

• Implementation of a management information system to facilitate program 
planning and evaluation; 

• Program schedules; 

• Hours of instruction; 

• Student recruitment; 

• Number of program staff; teacher/student ratio; 

• Education and experience of program staff; 

• Methods of class instruction/teaching strategies (e.g., lecture, small group, 
individualized, computer-assisted instruction); 
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• Curriculum materials; 

• Other support services provided by the program; 

• Linkages with vocational education and JTPA programs to coordinate basic 
education with vocational education and job training; and 

• Program revenues and expenditures (salaries, fringe benefits, travel, supplies, 
materials, and equipment). 

Participant Outcomes 

• Student participation/attendance (measured by hours of attendance); and 

• Student academic performance (measured by gains on TABE and completion of 
GED Tests). 

Selection of Measures 

Student attendance and academic performance were the two primary indicators chosen to 
evaluate program success. Students' hours of participation were determined based on class 
attendance records. Students with permanent record files were matched with students appearing 
on the class attendance records; these data were entered into a computer database and merged 
with data from permanent repords for analysis. Gains in educational achievement were measured 
by comparing the test scores on the TABE for all students with both pretest and posttest data on 
their cumulative record files. 

Characteristics of participants were drawn from program records of adults participating in 
the eight sample schools. Information on program costs was obtained from the survey of program 
coordinators and budget records provided by adult education coordinators from two of the islands. 
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Longitudinal Evaluation of ABE in Virginia, 1970-1985 

Evaluation Overview 

Virginia Commonwealth University conducted a longitudinal statewide evaluation of ABE 
programs in Virginia, comparing data collected in 1970, 1978, and 1985. The primary objectives 
of the evaluation were to appoint a statewide ABE evaluation advisory committee, design and 
implement a statewide program evaluation, conduct a representative statewide assessment of each 
school districts ABE students, and conduct a computer analysis of statewide data that were 
collected. The data from the 1985 evaluation were collected through survey instruments mailed 
to all state department ABE personnel, local ABE supervisors, learning center coordinators, and 
ABE counselors, as well as a random sample of paraprofessional ABE personnel, teachers, and 
students (the overall return was 81 percent). The primarily summative evaluation was modeled 
after Stake s evaluation design. Instruments used in the study included questionnaires, interview 
guides, data check sheets, and data summary sheets for each population studied. Evaluation staff, 
ABE staff, and students participated in the evaluation process. 

A long list of recommendations came out of the evaluation, but the consensus of 
questionnaire and interview responses was that programs made "remarkable progress" despite 
limited funding. State and local programs are continuing to respond to many of these 
recommendations. The study is one of few longitudinal studies in adult education and provides a 
useful documentation of evaluation methodology. 

Measures of Program Quality 

A selected listing of indicators is as follows: 

Program Context 

• Procedure for identifying and ranking state funding needs; 

• Monitoring procedures and criteria (e.g., periodic ABE supervisor visits to special 
projects; criteria for assessing satisfactory/unsatisfactory progress; use of separate 
standardized evaluation design for use with teacher-training activities and special 
projects); 

• Ongoing contact with the public school system to identify high school dropouts for 
recruitment in ABE programs; 

• Establishment of a statewide adult education resource center/clearinghouse to 
provide technical assistance in student evaluation, materials, equipment selection, 
program research and evaluation, and other instructional and student services; and 

• Characteristics of program participants: age distribution, marital status, racial 
distribution, income level, home ownership, urbanicity, employment status, 
mobility, educational attainment, parents' educational attainment, native language, 
length of time separated from school prior to ABE enrollment, access to 
transportation, reasons for leaving public school 
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Program Process and Content 



• State department and local program collaboration in conducting needs assessments 
to determine interest and demand for the program, especially in non-parlicipalinu 
school divisions; 

Program Promotion 

• Existence of local advisory committee; 

• Acceptance and support from the local community (e.g., local school district); 

• Cooperation with local community agencies; 

• Aggressive recruitment efforts, such as use of community resources and mass 
media publicity (radio, television); assignment of recruitment duties to specific 
personnel; targeting populations least served; 

Staffing/Training 

• Aggressive hiring of blacks and women in supervisory positions (aids student 
curriculum relevance, recruitment, and retention); 

• Requirements for personnel: availability/encouragement of certification 
endorsement for adult education personnel; master's degree in adult education for 
full-time ABE personnel; nine hours of graduate coursework in adult education for 
part-time personnel; endorsement in guidance and/or master's degree for ABE 
counselors (as conditions of continued ABE employment); 

• Full-time teaching positions; 

• Employment of support personnel, aides, and counselors to conduct testing, 
counseling, keep track of student records; 

• Annual, statewide staff training plan; 

• Availability of a variety of training activities (e.g., independent study, formal 
courses, workshops, retreats, small group training kits); local ABE supervisors fund 
and conduct local orientation and training sessions regularly; local program 
participation in cluster training and regional workshops; 

• In-service training on testing and counseling techniques; 

• Regularly scheduled staff meetings; regular contact between teachers and 
supervisors; 
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Assessment/Support Services 

• Measurement of entry and exit competency levels of students; 

• Availability of student personnel services-such as counseling, individual testing, 
placement, and follow-up-to complement instruction and aid retention; 

Program Location/Schediile/Facilities 

<> Location of classes in adult settings such as community colleges and secondary and 
vocational-technical schools, rather than elementary schools; 

• Classroom facilities include adequate storage space, cabinets, closets; designated 
private areas for interviewing and counseling; 

• Program schedule with summer classes to promote continuity; flexible hours to 
accommodate student schedules; 

• Close awareness of student attendance patterns so that schedules can be changed 
if needed; 

Instruction 

• Emphasis on small group instruction rather than lecture format; use of audio-visual 
presentations and aids, field trips; 

• Supplementary instruction on learning skills such as note taking, listening, study 
habits, learning strategies; 

• Opportunity for students to receive literacy and job skills training in the same 
location or in close proximity; encourage on-site workplace literacy programs; 

• Use of adult instructional materials and methods; 
Participant Outcomes 

• Gains in daily/weekly attendance; 

• Reading and math achievement (changes in grade level in relation to time on task; 
depends in part on student educational level, e.g., beginning reading takes more 
instructional time than reading improvement); 

• Changes in student perceptions of subject difficulty (e.g., reading was ranked as 
less difficult in 1985 than in 1970, indicating that reading materials and instruction 
may have improved); 

• Current and overall student progress (as rated by program personnel); 
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• Follow-up on participants who are absent frequently or who leave the program. 



Selection of Measures 

The majority of indicators listed above summarize recommendations for improving ABE 
programs, based on changes in population characteristics, ABE personnel and student perceptions, 
and program characteristics charted over the course of the evaluation. Some indicators also 
represent descriptive data that provide a context for participant needs and program services. The 
evaluation report provides numerous tables with comparative data on these and other 
characteristics of programs, teachers, and participants. The significance of many of these 
indicators, or their relationship to program quality, is unclear. 
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Washington State 
Adult Basic Education Student Follovwip Evaluation 

Evaluation Overview 

This report profiles an evaluation of participation in Federally funded ABE and GED 
programs in Washington in 1989. The purpose of the study, which was begun in 1990, was to 
"provide a descriptive, qualitative analysis of student participation outcomes." The focus of the 
study was not to evaluate ABE and GED programs, but to examine what effect program 
participants' said the programs had on their lives. ABE and GED teachers were trained to 
conduct interview's with a sample of former students from 16 of the state's community college- 
based ABE programs. Students were identified from a stratified random sample (by race and 
rurality) of students in the ABE population (not including ESL students) who had completed a 
minimum of 12 program contact hours; were enrolled in an adult education program during the 
1989 quarter but not in the spring quarter of 1989; and left the program for any reason other 
than injury or illness. Initially, 386 students were targeted for interviews; 293 were eventually 
included. The overwhelming majority of students who were interviewed indicated that most areas 
of their lives, particularly in personal growth and development, had improved since they began 
attending an adult education program. 

Measures of Program Quality 

Four major areas related to participation outcomes were examined: motivation and 
achievement, job preparation, personal and family benefits, and program evaluation. 
Approximately 80 questions were asked related to student goals and outcomes in these areas. 

Program Context 

• Motivation to attend ABE classes 
Program Process and Content 

• Student perception of quality of ABE/GED program 
Participant Outcomes 

• Achieving educational and personal goals; 

• Improving reading, math, writing, spelling; 

• Improving confidence in ability to use basic skills; 

• Obtaining a GED/high school diploma; 

• Enrollment in a college or apprenticeship/job training program; 

• Competence in basic communication skills; 
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Competence in functional life skills; 



Increased self-esteem; 
Obtaining a job; 

Increasing earnings/job performance; 

Obtaining a better job; 

Reduction in receipt of public assistance; 

Increased reading of books, magazines, newspapers; 

Generally improved quality of, and satisfaction with, life; and 

Involvement in the community. 



Selection of Measures 

Desired outcome measures were included in a structured interview instrument, which was 
developed by project staff, a project advisory committee, and the Washington State Supervisor of 
Adult Education and Literacy Programs. The evaluation report does not specify how the specific 
outcome measures were determined, although it appears that, within the four general categories 
mentioned above, interview designers relied on general assumptions about the relationship 
between education and personal benefits and the impact of basic communication skills and 
functional life skills on seeking and finding employment; an interest in learning what motivated 
adults to enroll in adult education programs and continue their education; and a belief that 
participants have a "unique perspective on the relative strengths and weaknesses of education 
programs." 
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ACHE Evaluation of Community Based Literacy Programs (1988-89) 

Evaluation Overview 

The Association for Community Based Education (ACBE) conducted in-depth 
longitudinal evaluations of nine community-based adult literacy programs. The purpose of the 
evaluations was to "expand knowledge about community based adult literacy programs" and to 
"produce rigorous evidence" of the impacts these programs have on participants. In spring 1 C AS7, 
programs were chosen from about 450 literacy providers. ACBE mailed mini-grant 
announcements to each of these providers, but received only seven applications; of these, three 
were usable. An additional 14 applications were solicited, using a "convenience sample" reflecting 
variety in geographical region, racial/ethnic groups, urban vs. rural, years in operation, number 
served, tutorial vs. group instruction, and curriculum materials used. Each of the final nine 
programs involved was awarded a $3,000 mini-grant for participation in a year of preliminary 
activities that began in the fall of 1987, and each was awarded a $5,000 mini-grant for 
participating in the evaluation the following year. 

Measures of Program Quality 

Evaluators, with input from the nine programs, selected 10 participant outcomes for 
measurement: 

(1) Reading skills 

(2) Writing skills 

(3) Math skills 

(4) Oral language skills (ESL) 

(5) Reading activities outside of class/tutoring 

(6) Writing activities outside of class 

(7) Fostering of children's intellectual and academic development 

(8) Community activities and contributions 

(9) Self-esteem 

(10) Self-determination 

Selection of Measures 

ACBE conducted an extensive search to find instruments to measure these 10 outcomes. 
Staff considered 17 criteria for selecting instruments, including: reliability, validity, suitability for 
group administration to non-readers and non-writers, content that was familiar and interesting to 
adults, and relative quickness and ease in administering. Evaluation staff found suitable 
instruments for only five outcomes; ACBE stafffconsultants developed five more. Programs 
selected four to seven outcomes to measure at their sites. Examples of instruments that were 
chosen include: 

• Adult Basic Learning Examination (ABLE) for reading comprehension and math; 

• SRA Reading Index for reading comprehension; 

• California Achievement Tests (CAT) for writing skills; 
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♦ Basic English Skills Test (BEST) lor oral English proficiency of ESL learners; and 

• Culture-Free Self Esteem Inventory. 

ACBE developed instruments to measure outcomes for which well-developed or widely used 
instruments were cither unavailable or insufficient for the purposes of the study. These outcomes 
were outside reading and writing activities, fostering children's academic and intellectual 
development, community involvement, and self-determination. ACBE staff visited programs three 
times (fall, winter, spring) and measured outcomes each time (pre- mid-, and post-tests). 
Outcomes varied at the different sites. 

ACBE conducted an analysis to find predictors of dropout and gains, but found no strong 
predictors. The report suggests the following activities for future evaluations: refine and extend 
ACBE "psychometric" process; use case studies or ethnographic studies also; give technical and 
financial assistance to local programs so staff can develop and implement their own evaluation 
process and so learners can develop/use self-assessment and participatory research processes. 
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Guidebook for Effective Literacy Practice, 1984 
National Adult Literacy Project 

Evaluation Overview 

The national study examined literacy programs in six settings: states and LEAs, 
employment and training, community based, corrections, military, and postsccondary. A total of 
335 programs were nominated to be studied, based on programs known for success in recruitment, 
retention, and "results"; and based on a "holistic programmatic view" of eight program 
components. The survey design involved mailing a letter, survey, checklist of materials, envelope, 
stamps, and postcard to each nominee (225 of 335 were returned and 213 were analyzed on 
computer). The survey contained questions about program statistics, learner demographics, 
program operations, program components, retention, indicators of success, and instructional 
materials. 

Measures of Program Quality 

Programs are considered to be effective when they operate by some or all of the 
following principles or mechanisms: 

Program Context 

• Clarity about the overall goal and philosophy of instruction; defined program goals 
and objectives; 

Program Process and Content 

• Goals are developed for every program component to determine effectiveness in 
meeting them; 

• Utilize targeted radio and TV advertisements; 

• Intake -program staff help potential learners determine if a progiam is well-suited 
to their goals; important to personalize and individualize the process, motivate 
students, clarify needs and goals, make appropriate placements, and put them at 
ease; careful diagnosis of individual learner needs and development of individually 
tailored learning plans; 

• Learning objectives are tied to instructional methods, materials, and assessment 
strategies; 

• Procedures are in place for making students' goals a benchmark of success 
(translate into concrete skills, areas of knowledge, curricular strategies); testing 
procedure or instrument appropriate to verification of students' goals and 
combined with other means of assessment; 

• Programs provide frequent feedback to learners on progress and carefully 
document the process; 
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Instruction utilizes relevant, meaningful materials; 



• Evaluation -evaluate program effectiveness frequently in meeting goals in each of 
ihe component areas and use data to improve the program; develop evaluation 
expertise; design formative and summative evaluation instruments to measure 
program goals; create and implement unified system for collecting, analyzing, and 
using data for program improvement; and 

• C ounseling -dcvelop personal relationship with adults, use different counseling 
options, explore alternative support systems, train staff in counseling techniques; all 
staff play a part: trained counselors, support staff, teachers, other students; 
important to ask how structure of counseling delivery suits learner needs. 

Participant Outcomes 

• Intended learning outcomes and standards for judging success in achieving those 
outcomes are explicit; and 

• Follow-up-short- and long-term follow-up needs to involve: good rapport with 
students so staff know what is going on in their lives; updated, accurate files on 
students to help facilitate tracking of students; long-term follow-up procedure 
geared toward getting information about program's long-term impact on students. 

Selection of Measures 

Measures of program effectiveness were culled from survey responses and interviews at 
selected sites. The measures were categorized into eight broad program components: 

• Recruitment; 

• Orientation (present potential participant with overview of educational tracks, 
counseling, support systems, choice of instructors or classes, special program 
features, instructional modes, flexible scheduling, materials, locations); 

• Counseling; 

• Diagnostic testing; 

• Instructional methods and materials; 

• Assessment of learners; 

• Follow-up; and 

• Program e valuation (formal record keeping and internal monitoring of program 
operation). 
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Evaluating National Workplace Literacy Programs 

Evaluation Overview 

Sticht s paper expands on evaluation issues outlined in the rules and regulations for the 
National Workplace Literacy Program (NWLP). The paper summarizes the Federal Regulations 
criteria for well-designed workplace literacy programs and provides a framework with which 
programs can use these criteria to "meet the requirements of the rules and regulations governing 
the NWLP so that the Department of Education can determine the value of programs." 

Measures of Program Quality 

Program Context 

• Defines roles of partners; demonstrates active participation of all partners; and 

• Targets adults with inadequate skills aimed at new employment, career 
advancement, or increased productivity. 

Program Process and Content 

• Documents/demonstrates need for workplace literacy program; 

• Defines/outlines activities, measurable objectives based on overall goals; 

• Collects data that can be used for program improvement, data that shows cost 
benefits; 

• Provides training through an educational agency; 

• Uses curriculum materials designed for adults that reflect the needs of the 
workplace; 

• Measures literacy abilities through and pre- and post-assessment; measurement of 
literacy abilities should reflect content of what is being taught; 

• Utilizes and provide evidence of experienced, qualified staff; and 

• Shows strong relationship between skills taught and literacy requirements of actual 
jobs. 

Participant Outcomes 

• Improved workforce literacy abilities and improved productivity 

Converse better with co-workers 

Read and write job materials better 

Know more about the job, workplace, community 

103 



120 



Perform job tasks better 
Get to work on time more 

Make use of employee benefits more appropriately 
Contribute more to team efforts 
Job retention 
Job advancement 

Se lection of Measures 

Criteria for well-designed workplace literacy programs are drawn from the regulations for 
the NWLP. These criteria are used to illustrate how criteria for evaluating proposals for the 
NWLP can be used to report evaluations of programs. Evalualors are cautioned when trying to 
link literacy development and job productivity. Sticht indicates, "Unless a direct relationship to 
some indicator of productivity can be demonstrated in the design of the program, the program 
developer should not promise to improve that aspect of job productivity." However, literacy 
developments may have an indirect effect; for example, if access to education programs helps to 
boost employee morale, indicators of productivity such as increased attendance, less tardiness, and 
improved teamwork may improve. 
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Workplace Literacy Program Evaluations 



Evaluation Overview 

In this draft paper presented to the Work in America Institute in 19 C X), Mikulecky and 
D'Adamo-Wcinstein provide a review of recent evaluations of workplace literacy projects. 
Evaluations of a variety of manufacturing, military, and hospital programs are profiled, including 
Polaroid, Planters Peanuts, RJ. Reynolds, the U.S. Navy, Rockwell International, Onan 
Corporation, Texas Instruments, and Philadelphia Hospital and Health Care. The evaluation 
designs vary, hut reveal that evaluation data on program effectiveness are limited, and information 
on cost effectiveness is unavailable. 

Measures of Program Quality 

Program Process and Content 

• Company support for workplace literacy programs; 

• Demand expressed for workplace literacy programs: number of employees 
enrolled, waiting to enroll; and 

• Flexibility in modifying decisions to meet student needs. 
Participant Outcomes 

• Obtain a GED diploma; 

• Pre- and post- test scores on reading skills; 

• Anecdotal reports of participant progress, informal feedback from participants, 
employers, instructors, supervisors; 

• Improved union-management relations; 

• Ability to summarize job materials in their own words; 

• Becoming eligible for promotion; and 

• Becoming "better citizens of the community." 

In addition to these elements pulled from the evaluations, the review notes goals, 
resources, and learning methods where programs have revealed inadequacies: 

• Evaluation feedback during the planning stages of a workplace literacy program so 
that planners and participants can clarify goals; 

• Sufficient learner time to practice literacy, accomplish goals; 
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• Hiring instructors with knowledge of basic skills instruction, as well as some 
knowledge or expertise about workplace literacy program development; 

• Providing feedback on learner accomplishments; and 

• Availability of materials (or resources to develop them) that match the workplace 
literacy goals. 

Selection of Measures 

Mikulecky and D'Adamo note that a variety of workplace literacy program reports cite 
their programs as being "exemplar)'" or "effective," but for the most part lack the data to support 
such assessments. Instead, most evidence for participant gains in these evaluations is based on 
anecdotal reports from workers, supervisors, and instructors. Examples of tests used to measure 
participant gains include the TABE, ABLE, instructor tests, and employment performance tests. 
Noting the absence of outcome data, the authors present a model for programs to provide 
summative and formative evaluation, evidence of goal attainment, and evaluation feedback 
through interviews, analy$is of memos and planning documents, and program observations. 
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Alaska 

AHE'JOBS Regional and Local Literacy Program Funding 



Summary' of Process 

Alaska utilizes a numerical rating system to review applications for ABE/JOBS 
Coordinated Funding as well as for its teacher training programs for regular ABE/JOBS programs 
and those in correctional institutions, LEAs» postsecondary institutions, CBOs and private 
training organizations (including profit making) are eligible recipients for funding. 

Criteria Reviewed 

Needs Assessment 

• Documented need 

• Use of supporting demographic data 

Program Activities 

• Numbers of students, sites, hours of instruction* 

• ABE program requirements (80 percent of students must have skill levels 
below 8th grade upon entry) 

Instructional Design 

• Life coping curriculum 

• Services to handicapped 

Assessment/counseling services 

• Child care and transportation availability 
Recruitment and Retention Strategies 

• Open entry/open exit programming 
Coordination of Programming Services 

• Letters of agreement outlining coordinated services, not just Wters of 
support 

• Broad based advisory councils/citizen planning sessions 

• JOBS services support 

♦Not necessary in teacher training applications 



108 



125 



Arizona 

Funding Criteria for ABE Programs 



Summary of Process 

Arizona gives first priority for funding to programs which have had projects in operation 
in the previous year and have met all their project objectives; second priority goes to programs 
which have been in operation but have met minimum project requirements; and a third priority 
focuses on expansion of new programs. Project funding is based on $150 per student enrolled for 
125 hours of instruction. Cost effectiveness is an important criteria for program funding. Projects 
are reviewed by the Adult Education Project Review Committee. Federal funds are primarily 
used to support least educated populations and programs with a minimum of 20 students per class. 
Funding is also available to programs which enroll less than 20 per class, but these programs arc a 
second priority. All activities described in the proposal must be supportive of the Arizona State 
Plan for Adult Education. 

Criteria Reviewed 

Meets State and Federal Rules and Regulations 

Needs Assessment 

• Provides activities to meet needs of local populations 

• Supports State plan objectives 

• Reaches least educated, most in need 

Coordination and Multiple Funding Support 

• Demonstrates resource sharing with JTPA, Title XX 

• Business and other governmental agencies 

• Focus on employability skills 

Outreach Efforts 

• Promotional program appropriate to population 

• Flexible scheduling and convenient locations 

• Availability of child care services 

Program Activities 

• Student tracking system 
Evaluation System for Teacher and Students 
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Staff Development Plan 

• Required Adult Education Certificate for teachers 
Budget Feasibility 
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Colorado 
Evaluation of ABE Proposals 



Summary of Process 

Colorado utilizes a formal proposal review process which recommends lull funding, partial 
funding or disapproval. Maximum points accrued: 100. 

Criteria Reviewed 

Needs Assessment (10 points) 

• Focus on special populations 

• Multiple service sites 

• Statistical data provided 

Instructional Program (30 points) 

• Identification of structure (homebound, tutorial, class) 

• Measurable and attainable goals supportive of State Plan 

• Objectives support needs assessment 

• Evaluation instruments described and appropriate to program objectives 

• Goals reflect improvement, expansion and innovation 

Program Plans Relating to Administrative Structure (10 points) 

• Staff development 

• Recruitment of volunteers 

• Accurate record keeping 

Staff Development Activities (15 points) 

• Plans for pre- and in-service training of professionals and volunteers 
Outreach Efforts (10 points) 

• Promotional system 

• Flexible scheduling and appropriate locations 
Cooperative Arrangements (15 points) 

• Evidence of planning, support and commitment from cooperating agencies 
Budget including Supplemental Funding (5 points) 

Previous Experience of Applicant (5 points) 
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Delaware 

Criteria for Evaluating ABE Funding Proposals 



Summary of Process 

Delaware has developed a numerical rating scale which staff of the State Education 
Agency utilizes to determine funding eligibility. Preference is given to those applicants who have 
demonstrated or can demonstrate a capability to recruit, train and serve educationally 
disadvantaged adults. Criteria reflect issues of need, capability and commitment to serve most in 
need, program design, effectiveness of applicant in providing services, community coordination 
and evaluative processes. 

Criteria Reviewed 

Assessment of Need (5 points) 

Measurable Objectives Relating to Priority Areas of Service (5 points) 
Personnel Qualifications (10 points) 
Staff Development Plan (10 points) 
Recruitment Plan (15 points) 

• Numbers of students to be served 

• Ability to recruit educationally disadvantaged 

• Met recruitment goals from previous year 

Instructional Program (15 points) 

• Use of student learning plan 

• Use of diagnostic/prescriptive approach 

• 100 hours plus student instruction 

• Use of appropriate assessment system 

Retention Plan (15 points) 
Coordination Activities (10 points) 
Student Outcomes/Evaluation Plan (10 points) 
Budget Plan (5 points) 
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Iowa 

Criteria for Funding of ABE/ASE Programs 



Summary of Process 

Applications for program funding are reviewed by state personnel staff and members 
the State Advisory Council An evaluation checklist is included in the Iowa State Plan. No 
information as to rationale for these criteria or value attached to each was included. 



Criteria Reviewed 

• Staff development activities 

• Facilities and locations of programs 

• Hours of services 

• Progress in meeting objectives and expanding on them from year to year 
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Kansas 

Criteria for Evaluation of ABE Program Applications 



Summary of Process 

A formal checklist is utilized with ycs/no columns. No relative values are indicated for 
these criteria, nor is there indication of the rationale for selecting these criteria. 

Criteria Reviewed 

Needs Assessment Data 

• Program emphasis: ABE/ASE/training^related 

• Targeted areas: rural urban, high unemployment 

• Handicapped, homeless, etc. 

Program Activities 

• Flexible schedules 

• Convenient locations 

• Availability of support services 

• Reflects State adult education priorities 

Program Implementation 

• Activities geared to identified needs 

• Goals and objectives that support expansion of services 

Program Planning 

• Ongoing consultation and coordination with other agencies in planning, service 
delivery 

Resources 

• Adequacy of other sources of support to meet program requirements 
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Maine 

Criteria for Review of ABE Program Applications 



Summary of Process 

Maine provides a worksheet with values attached to various areas of the project proposal 
for review. No indication is given as to who does the reviewing, or rationale for criteria selected. 
Total numerical value to be attained in 100 points; no cutoff is provided. 

Criteria Reviewed 

Needs Assessment (10 points) 

• Use of demographic information 

• Attention to special needs (instructional services and support services to support 
client population) 

Program Planning/Cooperative Arrangements (8 points) 

• Clear planning process 

• Demonstrated involvement of other agencies 

• Relationship of ABE to other social services 

Program Objectives (50 points) 

• Clear, measurable objectives 

• Numbers, geographic area to be served 

• Service to least educated and special populations including homeless, incarcerated 

• Activities and time schedule sufficient to accomplish program goals 

• Flexible scheduling and appropriate locations for activities 

• Use of assessment and diagnosis for placement 

• Individualization of instruction 

• Counseling, referral and placement services available 

• Provision for outreach, continuous recruitment and open enrollment 

Staffing (8 points) 

• Qualified staff available 
Evaluation (10 points) 

• Process described 
Facilities and Materials (4 points) 

• Accessibility for handicapped 
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Budget (10 points) 

♦ Complete submission, including local resources and requirement that 95% of grant 
be used for instructional purposes 
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Maryland 

Funding Criteria for ABE Program Grants 
For LEAs and Community Colleges 



Summary of Process 

Maryland requires written narratives focused on the following areas: measurable 
objectives based on priorities in Maryland's State Plan for Adult Education, operational plan, key 
personnel, with particular attention to changes in staffing and the program s response to any 
evaluation recommendations made by the State in the previous year. This process is utilized lor 
programs which have approved multi-year plans. Proposals must demonstrate State s commitment 
to reaching adult populations most in need of, and least likely to participate, in adult education 
and must demonstrate coordination of resources and services. 

Criteria Reviewed 



Measurable Objectives (30 points) 



# 



Numbers to be served 
Outcomes anticipated 
Specified areas of priority and recruitment 



Operational Plan (30 points) 



Location and schedules of programs 
Total hours of instruction 
Instructional designs 
Recruitment plans 



Personnel (10 points) 



Implementation of Evaluation Recommendation (15 points) 



Annual Budget and Cost Effectiveness (15 points) 
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Montana 

Criteria for Review of ABE Project Applicants 



Summary of Process 

A formalized review form is utilized by the Montana Department of Public Instruction. 
Up to 100 points may be awarded to each proposal. Two individuals (one of whom may be an 
adult education specialist) are asked to review and rate proposals. This process is utilized for 
ABE program grants, teacher training projected and other special project applications. No 
information is provided as to a cut-off point at which a proposal is not acceptable, nor is any 
rationale provided for the selection of criteria. 

Criteria Reviewed 

Statement of Need (22 points) 

• Local demographics included 

• Coordination with needs of business/industry in area 

• Mutual cooperation with other appropriate agencies 

• Statement clear and concise 

Objectives of Plan (23 points) 

• Number of participants 

• Descriptions of programs 

• Description of program impact on participants 

Activities Supporting Program Objectives (10 points) 

• Description of activities 

• Description of data collection process 

• Evaluation relevant to objectives 

• Quantifiable measures 

Evaluation Plan (23 points) 

• Number of participants 

• Descriptions of programs 

• Description of program impact on participants 

Outreach Efforts (10 points) 

• Advertising plan in place with formal and informal methods to reach public 

• Focus on adults with lowest academic levels 
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Budget and Narrative Information (20 points) 

♦ Budget supportive of objectives 

♦ Local match identified 

♦ Budget reflective of population density and supportive for satellite programs 
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Oklahoma 

Criteria for Review of Project Applicants 



Summary of Process 

Oklahoma utilizes a checklist to evaluate funding proposals for general ABE/ASE 
programs, ABE Special Projects and Teacher Training Grants. The value assigned to each jf the 
applicable criteria is determined by the State. No information was provided as to relative value of 
the following criteria nor rationale for how they were selected. 

Criteria Reviewed 

General Criteria for All Proposals 

Clearly stated objectives 
Sound operational plan 
Activities are relevant and necessary 
Qualified personnel 

Adequate facilities and resources to carry out plan 
Adequate evaluation provisions to determine program effectiveness 

Special Projects 

• Innovative methods, systems and materials 

• Cooperative efforts with other Federal, State or local programs, business, industry, 
labor to promote coordinated activities and eliminate duplicative efforts 

• Addresses critical educational needs 

• Development of concepts, techniques and practices which are transferable 

• Will strengthen adult education state delivery system 

Teacher Training Grants 

Supports State Plan objectives for staff development activities 
Utilizes innovative methods, systems, materials or program 
Supports local needs 

Addresses critical training needs identified at national or state level 
Provides for program evaluation 

Supports other demonstration projects in geographic area 
Coordinates with other state staff development efforts 

Coordinates with other Federal assisted programs, including cooperative efforts 
with business and labor 
Includes eligibility criteria for participation 
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Wisconsin 

Criteria for Review of ABE Project Applications 



Summary of Process 

Wisconsin utilizes a numerical rating scale for review, (maximum of 100) with ranges from 
17 - 2(1 Excellent; 13 - 16, Very Good; 8 - 12, Good; 4 - 7, Fair; and 1 - 3, Poor. A rating of 
excellent implies that all requirement for the program have been exceeded; very good implies that 
all program requirements have been met; good indicates that most of the program goals could be 
accomplished; fair indicates inadequacies in several areas and a rating of poor indicates 
deficiencies in many program areas and a decision thai the proposal is incapable of accomplishing 
the program objectives. No indication is given as to who does the evaluating or the rationale for 
the criteria selected. 

Criteria Reviewed 

Needs Assessment (20 points) 

• Documented need 

• Use of supporting data 

Coordination and Multiple Funding Support (15 points) 

• Identification of other sources and purposes of funding 

• Evidence of collaborative planning/coordination of services 

• Identification of services to minorities and handicapped students and discussion of 
transition services into vocational programs (Wisconsin's program is managed by 
the Vo-Tech Post-Secondary System) 

• Measurable short and long term goals 

• Plan reflects recommendations of state agency review team 

Program Activities (30 points) 

• Activities support needs identified 

• Specified numbers of participants 

• Adequate staffing 

• Budget is reflective of program activities 

Evaluation Activities (15 points) 

• Evaluation process designed to measure impact 

• Appropriate use of staff, participants, administrators and advisory committee 
members for evaluation 

• Adequate provision for participant data collection 
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Program Scheduling (5 points) 

• Adequate and logical sequence for program activities 
Budg et (15 points) 

• Figures adequate and reasonable for program activities 
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Additional States 



The following states indicated iY ziv criteria for funding as follows: 
Alabama 

• Numbers of students served 

• Services to distinct populations (low-lcvcl, rural) 

New Hampshire 

• Program goals 

• Planning and coordination 

• Numbers of students served 

• Instructional time frame 

• Budget/Cost-effectiveness 

• Staff qualifications 

• Support services 

Idaho 

• Numbers of clients served - ABE/ASE 

• Outreach efforts 

• Needs assessment 

• Program planning 

• Measurable goals and objectives 

• Staff qualifications 

• Inservice training plan 

• Evaluation component 

Oregon 

• Goals and objectives supporting state goals 

• Numbers of clients served 

Tennessee 

• Coordination/linkages with other programs 

• Past experience in delivery services 

• Accountability for use of funds 

• Staff Qualifications 

• Commitment to state goals 

• Measurable goals and objectives 
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